← Back to Blog Home

Distributing Large Files Easily with Amazon S3

Developer Tools
by: Mike Swieton

Article summary

I’ve found it very useful to use Amazon’s S3 to distribute large files to customers — not just as an application back end, but as a simple way of distributing and archiving data. This is not at all surprising or innovative, as S3 is for serving files, but it didn’t seem obvious to me how to set it up a simple file distribution area.

Eventually, I boiled it down to:

$ rake s3:upload[MyFile.zip]
$ rake s3:urls

https://s3.amazonaws.com/MyBucket/MyFile.zip?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Expires=1373635405&Signature=GasdfTHxYCBwhH%2FhyktUD0RuI8o%3D

The first line uploads a file, and the second line will produce a signed link to that file that anyone can use for a limited time (until the Expires time). This was finally a workflow I could use.

Read more to see a complete example of how to do it.

Why S3?

Why not Dropbox? Or Basecamp? Or… XYZ? Honestly, it’s partly because Atomic has an S3 account already. Additionally, S3 storage is very cheap if you can wrangle the user-interface, which is very flexible (much more than is needed for simple distribution).

1. Setting up S3 & Permissions

Prerequisites

Obviously, you’ll need an Amazon Web Services account. The sample code listed below is for Rake (and is written in Ruby). And finally this wraps Tim Kay’s fantastic aws script.

Find User Id

Amazon provides fine-grained access management through IAM (Identity & Access Management). From the AWS console, you can get to the IAM pages, and from there you can create a user and sign keys. I won’t cover that part here, as it’s well documented elsewhere.

With an IAM user setup, click on the user and view the Summary tab. Under User ARN, you’ll see a value that looks like `arn:aws:iam::123456789012:user/ausername`. Copy that somewhere so we can get at it in the next step.

Bucket & Policy Setup

The part that confused me most was setting up S3 with a bucket and security. First, create a new bucket. In this documentation I’ve called it `ao-demo-bucket`. Next, select the newly-created bucket, go under *Properties* -> *Permissions*, and and click *Add bucket policy*. You should see a dialog with a text box. The simplest way to start is to use their policy generator, so next click that link on the editor dialog.

In the generator, select the policy type of *S3 Bucket Policy*. Then do the following:

  1. For the principal, use the “User ARN” that you copied earlier.
  2. Select S3 as the service.
  3. Select *All Actions*. The Amazon Resource Name should refer to your bucket, and it will look like arn:aws:s3:::.
  4. Click *Add Statement*. You’ll see it added to a list at the bottom of the form.

Next, we’ll add another statement to this policy.

  1. Enter the *User ARN* again as the principal.
  2. Select S3 as the service again.
  3. Select *All Actions*.
  4. Enter an Amazon Resource Name that refers to the *contents* of your bucket, looking like: arn:aws:s3:::/*. Note the trailing “/*” in the principal.
  5. Click *Add Statement* again.

We’re almost there! You should have a policy that looks something like:

{
  "Id": "Policy1373378044308",
  "Statement": [
    {
      "Sid": "Stmt1234567890123",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::ao-demo-bucket",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/ausername"
        ]
      }
    },
    {
      "Sid": "Stmt1234567890123",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::ao-demo-bucket/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/ausername"
        ]
      }
    }
  ]
}

Take that and paste it into the bucket policy editor back in the AWS console.

2. Upload Files & Distribute Links

Now, we need to get files up there. Take Tim Kay’s aws script and place it in your path somewhere.

Here’s the Rakefile I’ve been copying around to different projects. Just update the configuration section with your own authentication keys and bucket name (and the path to the `aws` script if necessary), and you’re set.

require 'openssl'
require 'cgi'
require 'base64'

namespace :s3 do
  ## CONFIGURATION ##############################################
  AWS_BIN = "aws"

  AWS_ACCESS_KEY_ID="MY_ACCESS_KEY_ID1234"
  AWS_SECRET_ACCESS_KEY="MY_SECRET_KEY123456789012345678901234567"

  # Links generated by the s3:urls task will expire in this many seconds.
  EXPIRATION_TIME = (3600*72) # 3 days

  BUCKET = "ao-demo-bucket"
  
  ## TASKS ######################################################

  ENDPOINT="https://s3.amazonaws.com/#{BUCKET}"

  def aws
    "env AWS_SECRET_ACCESS_KEY=#{AWS_SECRET_ACCESS_KEY} AWS_ACCESS_KEY_ID=#{AWS_ACCESS_KEY_ID} #{AWS_BIN} "
  end

  desc "List the contents of the bucket"
  task :ll do
    sh "#{aws} ls -l #{BUCKET}"
  end

  desc "List the contents of the bucket"
  task :ls do
    sh "#{aws} ls -1 #{BUCKET}"
  end

  desc "Remove a file"
  task :rm, :filename do |t, args|
    filename = args[:filename].to_s.strip
    raise ArgumentError, "filename argument is required" if filename == ""
    sh "#{aws} rm #{BUCKET}/#{filename}"
  end

  desc "Upload a file"
  task :upload, :filename do |t, args|
    filename = args[:filename].to_s.strip
    raise ArgumentError, "filename argument is required" if filename == ""
    filename = File.expand_path(filename)
    sh "#{aws} put --progress #{Shellwords.escape("#{BUCKET}/#{File.basename(filename)}")} #{Shellwords.escape(filename)}"
  end

  desc "Upload an OVA to S3 with a date"
  task :publish, :filename do |t, args|
    filename = args[:filename].to_s.strip
    raise ArgumentError, "filename argument is required" if filename == ""
    filename = File.expand_path(filename)
    ts = DateTime.now.strftime("%Y-%m-%d-%H:%M:%S")

    # Attempt to add a timestamp to the filename before the extension. This doesn't attempt to handle corner cases
    dest = File.basename(filename.sub(%r|(.[^.]+)$|, "-#{ts}1"))
    sh "#{aws} put --progress #{Shellwords.escape("#{BUCKET}/#{dest}")} #{Shellwords.escape(filename)}"
  end

  desc "Run an arbitrary aws script command"
  task :sh, :cmd do |t, args|
    sh "#{aws} #{args[:cmd]}"
  end

  desc "Generate a signature"
  task :urls do
    expires = Time.now.to_i + EXPIRATION_TIME
    digest = OpenSSL::Digest::Digest.new('sha1')

    puts "[expiration]: [url]"

    lines = `#{aws} ls -1 #{BUCKET}`.split(/$/).map(&:chomp).reject(&:empty?)
    lines.each do |line|
      file = line.strip
      content = "GETnnn#{expires}n/#{BUCKET}/#{file}"
      signature = CGI.escape(Base64.encode64(OpenSSL::HMAC.digest(digest, AWS_SECRET_ACCESS_KEY, content)).chomp)
      puts "#{Time.at(expires)}: #{ENDPOINT}/#{file}?AWSAccessKeyId=#{AWS_ACCESS_KEY_ID}&Expires=#{expires}&Signature=#{signature}"
    end
  end
end

The `s3:urls` task uses a SHA1 HMAC signature to sign the link. This way, we know it comes from an authorized user (you), and the filename and expiration dates cannot be changed without invalidating the signature. You can then distribute links easily and know that they will only be available for a limited time to people that have the link. It isn’t perfect security, but it’s very easy to use for both the person distributing the files and the person downloading them.

3. Tying it All Together

If everything’s configured right, you should be able to run some commands like these:

rake s3:ls                # List all the files that've been uploaded
rake s3:ll                # List all the files that've been uploaded like "ls -l"
rake s3:rm[filename]      # Remove a file from the bucket
rake s3:upload[filename]  # Upload a file
rake s3:publish[filename] # Upload a file and add a timestamp to the filename (myFile.zip -> myFile-2013-07-02-07:47:33.zip)
rake s3:urls              # Generate time-limited URLs to the contents of the bucket

 

Related Posts