14 Comments

Making AWS More Affordable with EC2 Scheduling

Amazon Web Services (AWS) provides an amazingly flexible platform for running and managing virtual machines in their Elastic Compute Cloud (EC2). With EC2, it is almost effortless to spin up clusters of dozens to hundreds of nodes. This allows for incredible flexibility in setting up various environments, for purposes such as development, testing, and production.

Of course, all of this comes with the hourly cost of running EC2 instances. Some EC2 instances, such as Windows instances, cost a substantial amount per month. While it probably won’t break the bank, it certainly factors into the decision of how many nodes and environments can be spun up and kept active.

The prevailing attitude often seems to be that EC2 instances must be kept running 24/7. This seems to ignore one of the great attractions of EC2 (and other AWS services) — you only pay for the resources that you consume. Keeping an instance running 24/7 when it isn’t actually being utilized is consuming unnecessary resources. Turning off instances when they won’t be utilized eliminates this resource wastage, and reduces cost. Fortunately, EC2 has a powerful set of tools that makes that very easy to configure schedules for turning instances on and off, and re-assigning static IP addresses.

Cost Savings Potential

Let’s say you have an application that needs to run on Windows IIS 8.0 and has a fairly large database which you’d like to keep in RAM (and can be all run on a single instance). You might need a m1.medium EC2 instance. With on-demand pricing, this costs approximately $133/month. If you need development, testing, QA, and production environments, this quickly becomes fairly expensive ($532/month). However, if you turn off the non-production environments (development, testing, and QA) when they won’t be used (such as at night), you can greatly reduce the total cost of the operation:

100% Utilization Production: 24/hours day — $133/month QA: 24/hours day — $133/month Testing: 24/hours day — $133/month Development: 24/hours day — $133/month Total: $532/month

Selective Utilization Production: 24/hours day — $133/month QA: 12/hours day — $67/month Testing: 12/hours day — $67/month Development: 12/hours day — $67/month Total: $334/month

This reduces the cost from $532/month to only $334/month — a significant amount. The savings could be substantially higher for larger or more resource-intensive environments. You can use the the AWS Simple Monthly Calculator to easily estimate maximum cost, and potential savings.

EC2 Scheduling

There is no built-in method to configure schedules for EC2 instances. However, the tools exist to readily create your own. The only requirement is a separate, continuously-running machine with Java that can send the necessary commands to EC2. Generally, this is easiest to do on a Linux box.

The EC2 API Tools

AWS provides a set of EC2 API tools to interact with EC2 resources. These can be downloaded directly from AWS, or found in popular Linux package management repositories. (Though these may be somewhat out of date). You’ll need these installed and configured in order to set up a schedule. On a Mac, you can install the tools with the ec2-api-tools formula in Homebrew.

AWS Credentials and Policies

In order to interact with EC2 resources, you’ll need to have an AWS access key and secret key. These serve to identify you to AWS. In addition, your account must have the necessary permissions to run certain commands on EC2 resources.

Because the EC2 tools will be run in an automated manner, it would be best to create a specific user for this purpose. Then generate an access key and secret key for the user. You should grant the user only limited permissions: the ability to turn on and off EC2 instances, and associate an Elastic IP addresses.

The following IAM policy allows the grantee to do this for all EC2 instances:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt123456789",
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:AssociateAddress"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

This IAM policy can be applied to the user specifically, or to a group which the user is a member of. The AWS documentation on applying IAM policies is very thorough.

The Commands

There are three main commands that will be utilized for scheduling: ec2-start-instances, ec2-stop-instances, and ec2-associate-address. These commands presume that you already have EC2 instances provisioned, and that they have been associated with Elastic IP’s. The commands require the EC2 ID of the instances, and the Elastic IP addresses (these can be located in the EC2 console).

Stopping EC2 Instances:

ec2-stop-instances <instance_ids>

e.g. ec2-stop-instances i-912345ab i-812345bc i-712345cd

Starting EC2 Instances:

ec2-start-instances <instance_ids>

e.g. ec2-start-instances i-912345ab i-812345bc i-712345cd

Associating Elastic IP Addresses:

ec2-associate-address <elastic_ip> -i <instance_id>

e.g.

ec2-associate-address 54.224.200.100 -i i-912345ab

ec2-associate-address 54.224.230.125 -i i-812345bc

ec2-associate-address 54.224.210.133 -i i-712345cd

Note: It is necessary to re-associate the Elastic IP address after instance start as the Elastic IP addresses associations are removed when an instance is stopped.

These commands can be run as-is if the AWS access key and secret key are provided in the environment variables AWS_SECRET_KEY and AWS_ACCESS_KEY; otherwise these parameters must be specified on the command line:

e.g. ec2-stop-instances --aws-access-key KEY --aws-secret-key KEY i-912345ab

The Schedule

These discrete commands make it very easy to use cron or other time-based job schedulers to arrange a scheme for stopping and starting EC2 instances.

For instance, you can have all of your non-production EC2 instances shut down at 7PM, and start again at 7AM:

0 7 * * * /bin/bash -c ~/start_instances.sh

# start_instances.sh
 
export AWS_ACCESS_KEY="UTHEENDZNAPPA20HAI11"
export AWS_SECRET_KEY="eij5ugaiphee6uusheg6eiVaiD0ein8moh7ieS0o"
 
/usr/local/bin/ec2-start-instances i-912345ab i-812345bc i-712345cd
sleep 5
/usr/local/bin/ec2-associate-address 54.224.200.100 -i i-912345ab
/usr/local/bin/ec2-associate-address 54.224.230.125 -i i-812345bc
/usr/local/bin/ec2-associate-address 54.224.210.133 -i i-712345cd

0 19 * * * /bin/bash -c ~/stop_instances.sh

# stop_instances.sh
 
export AWS_ACCESS_KEY="UTHEENDZNAPPA20HAI11"
export AWS_SECRET_KEY="eij5ugaiphee6uusheg6eiVaiD0ein8moh7ieS0o"
 
/usr/local/bin/ec2-stop-instances i-912345ab i-812345bc i-712345cd

The stop_instances.sh script contains appropriate ec2-stop-instances calls, and the start_instances.sh script contains the appropriate ec2-start-instances calls followed by the appropriate ec2-associate-address calls. Sometimes it is necessary to sleep briefly after starting an instance in order to wait for the instance to be in a state in which an Elastic IP address can be associated.

Conclusion

While you often don’t have the flexibility to coordinate a schedule for EC2 instances (production has to be available 24/7), the AWS platform is flexible enough to allow you to program a schedule that makes sense for your particular environment(s). Being able to tune your EC2 instances to only consume AWS resources when they are actually utilized is more efficient, and prevents unnecessary expenses. When considering specialized or especially resource intensive environments, this can be of substantial benefit.