Making AWS More Affordable with EC2 Scheduling

Amazon Web Services (AWS) provides an amazingly flexible platform for running and managing virtual machines in their Elastic Compute Cloud (EC2). With EC2, it is almost effortless to spin up clusters of dozens to hundreds of nodes. This allows for incredible flexibility in setting up various environments, for purposes such as development, testing, and production.

Of course, all of this comes with the hourly cost of running EC2 instances. Some EC2 instances, such as Windows instances, cost a substantial amount per month. While it probably won’t break the bank, it certainly factors into the decision of how many nodes and environments can be spun up and kept active.

The prevailing attitude often seems to be that EC2 instances must be kept running 24/7. This seems to ignore one of the great attractions of EC2 (and other AWS services) — you only pay for the resources that you consume. Keeping an instance running 24/7 when it isn’t actually being utilized is consuming unnecessary resources. Turning off instances when they won’t be utilized eliminates this resource wastage, and reduces cost. Fortunately, EC2 has a powerful set of tools that makes that very easy to configure schedules for turning instances on and off, and re-assigning static IP addresses.

Cost Savings Potential

Let’s say you have an application that needs to run on Windows IIS 8.0 and has a fairly large database which you’d like to keep in RAM (and can be all run on a single instance). You might need a m1.medium EC2 instance. With on-demand pricing, this costs approximately $133/month. If you need development, testing, QA, and production environments, this quickly becomes fairly expensive ($532/month). However, if you turn off the non-production environments (development, testing, and QA) when they won’t be used (such as at night), you can greatly reduce the total cost of the operation:

100% Utilization
Production: 24/hours day — $133/month
QA: 24/hours day — $133/month
Testing: 24/hours day — $133/month
Development: 24/hours day — $133/month
Total: $532/month

Selective Utilization
Production: 24/hours day — $133/month
QA: 12/hours day — $67/month
Testing: 12/hours day — $67/month
Development: 12/hours day — $67/month
Total: $334/month

This reduces the cost from $532/month to only $334/month — a significant amount. The savings could be substantially higher for larger or more resource-intensive environments. You can use the the AWS Simple Monthly Calculator to easily estimate maximum cost, and potential savings.

EC2 Scheduling

There is no built-in method to configure schedules for EC2 instances. However, the tools exist to readily create your own. The only requirement is a separate, continuously-running machine with Java that can send the necessary commands to EC2. Generally, this is easiest to do on a Linux box.

The EC2 API Tools

AWS provides a set of EC2 API tools to interact with EC2 resources. These can be downloaded directly from AWS, or found in popular Linux package management repositories. (Though these may be somewhat out of date). You’ll need these installed and configured in order to set up a schedule. On a Mac, you can install the tools with the ec2-api-tools formula in Homebrew.

AWS Credentials and Policies

In order to interact with EC2 resources, you’ll need to have an AWS access key and secret key. These serve to identify you to AWS. In addition, your account must have the necessary permissions to run certain commands on EC2 resources.

Because the EC2 tools will be run in an automated manner, it would be best to create a specific user for this purpose. Then generate an access key and secret key for the user. You should grant the user only limited permissions: the ability to turn on and off EC2 instances, and associate an Elastic IP addresses.

The following IAM policy allows the grantee to do this for all EC2 instances:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt123456789",
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:AssociateAddress"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

This IAM policy can be applied to the user specifically, or to a group which the user is a member of. The AWS documentation on applying IAM policies is very thorough.

The Commands

There are three main commands that will be utilized for scheduling: ec2-start-instances, ec2-stop-instances, and ec2-associate-address. These commands presume that you already have EC2 instances provisioned, and that they have been associated with Elastic IP’s. The commands require the EC2 ID of the instances, and the Elastic IP addresses (these can be located in the EC2 console).

Stopping EC2 Instances:

ec2-stop-instances

e.g. ec2-stop-instances i-912345ab i-812345bc i-712345cd

Starting EC2 Instances:

ec2-start-instances

e.g. ec2-start-instances i-912345ab i-812345bc i-712345cd

Associating Elastic IP Addresses:

ec2-associate-address -i

e.g.

ec2-associate-address 54.224.200.100 -i i-912345ab

ec2-associate-address 54.224.230.125 -i i-812345bc

ec2-associate-address 54.224.210.133 -i i-712345cd

Note: It is necessary to re-associate the Elastic IP address after instance start as the Elastic IP addresses associations are removed when an instance is stopped.

These commands can be run as-is if the AWS access key and secret key are provided in the environment variables AWS_SECRET_KEY and AWS_ACCESS_KEY; otherwise these parameters must be specified on the command line:

e.g. ec2-stop-instances --aws-access-key KEY --aws-secret-key KEY i-912345ab

The Schedule

These discrete commands make it very easy to use cron or other time-based job schedulers to arrange a scheme for stopping and starting EC2 instances.

For instance, you can have all of your non-production EC2 instances shut down at 7PM, and start again at 7AM:

0 7 * * * /bin/bash -c ~/start_instances.sh

# start_instances.sh

export AWS_ACCESS_KEY="UTHEENDZNAPPA20HAI11"
export AWS_SECRET_KEY="eij5ugaiphee6uusheg6eiVaiD0ein8moh7ieS0o"

/usr/local/bin/ec2-start-instances i-912345ab i-812345bc i-712345cd
sleep 5
/usr/local/bin/ec2-associate-address 54.224.200.100 -i i-912345ab
/usr/local/bin/ec2-associate-address 54.224.230.125 -i i-812345bc
/usr/local/bin/ec2-associate-address 54.224.210.133 -i i-712345cd

0 19 * * * /bin/bash -c ~/stop_instances.sh

# stop_instances.sh

export AWS_ACCESS_KEY="UTHEENDZNAPPA20HAI11"
export AWS_SECRET_KEY="eij5ugaiphee6uusheg6eiVaiD0ein8moh7ieS0o"

/usr/local/bin/ec2-stop-instances i-912345ab i-812345bc i-712345cd

The stop_instances.sh script contains appropriate ec2-stop-instances calls, and the start_instances.sh script contains the appropriate ec2-start-instances calls followed by the appropriate ec2-associate-address calls. Sometimes it is necessary to sleep briefly after starting an instance in order to wait for the instance to be in a state in which an Elastic IP address can be associated.

Conclusion

While you often don’t have the flexibility to coordinate a schedule for EC2 instances (production has to be available 24/7), the AWS platform is flexible enough to allow you to program a schedule that makes sense for your particular environment(s). Being able to tune your EC2 instances to only consume AWS resources when they are actually utilized is more efficient, and prevents unnecessary expenses. When considering specialized or especially resource intensive environments, this can be of substantial benefit.

Conversation
  • Dax Fohl says:

    I’ve found that EC2 is good for things you need to spin up and down at will like my bamboo build agents, but beyond that, well this guy says it better than I could: http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/

  • Stephan says:

    For those who are not familiar with with scripts I have written an easy to use windows tool that allows you to schedule starts/stops/backups of EC2 and RDS instances:

  • jay chapel says:

    This is a very helpful post, thank you. I’d love everyone’s feedback on a web app we launched recently that enables automated scheduled on/off times for AWS instance. See http://www.parkmycloud.com. Thanks!

  • Tonderai@aws says:

    The AWS Data Pipeline is uniquely suited to this task. Data Pipeline uses AWS technologies and can be configured to run AWS CLI commands on a set schedule with no external dependencies. Data Pipeline can write logs to S3 and runs in the context of an IAM role, which eliminates key management requirements. Data Pipeline is also cost effective; for example, the Data Pipeline free tier can be used to stop and start instances once per day.

    For more details: https://aws.amazon.com/premiumsupport/knowledge-center/stop-start-ec2-instances/

  • Hyip says:

    Automatically start and terminate the instance using Scheduling and Amazon API may cause data losses on that event. I’d recommend the Stop & Recover Actions using AWS CloudWatch Alarm

    For more details: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/UsingAlarmActions.html

  • Hide your access and secret key shown in the .sh examples from the post

  • I am using following python code in AWS Lambda for scheduling start and stop my servers. It got two options.
    1. You can invoke the Lamda function from a management server. With list of servers to be started or stopped. This method helps if your servers are running applications that needs to be shut-downed before powering off the servers.
    OR
    2. You can schedule the lambda function to run every hour. It will read your EC2 tags and will take necessary action.

    https://github.com/bijohnvincent/ScheduleStartStopEc2Lambda

  • Wow. That is so elegant and logical and clearly explained. Brilliantly goes through what could be a complex process and makes it obvious.

  • Kumar says:

    Can anyone tell me the automated procedure to take a RDS SNAPSHOT(before shutdown), then instance Shutdown, then Startup automatically at a scheduled time. Since the business do not afford the cost in non business hours for QA,Test databases.
    So is there an automated way to do this for RDS postgresql instances ? Please help me with this. Thanks.

  • Comments are closed.