We're hiring!

We're actively seeking designers and developers for all three of our locations.

Difficulties Uploading to Amazon S3 over SSL with EventMachine

I recently did some experimenting using EventMachine for uploading multiple files in parallel to Amazon S3. I’d heard about a non-blocking S3 client called happening on an episode of Ruby5 so I decided to give it a try.

I wrote a simple script that would upload a 22 mb file from my laptop to an S3 bucket. I put in a periodic timer that would print a “.” to the screen so I could verify that the main event loop was not being blocked during the upload.

require 'happening'

EM.run do 
  EventMachine::add_periodic_timer(1) { print "." }

  item = Happening::S3::Item.new('my_bucket', 
                                 "22mb_file", 
                                 :aws_access_key_id => 'MY_ACCESS_KEY', 
                                 :aws_secret_access_key => 'my_secret_key', 
                                 :permissions => 'public-read')
  item.put File.read("/tmp/22mb_file") do |response|
    puts "Finished"
    EM.stop
  end
end

I ran the script and watched the Activity Monitor so I could keep an eye on CPU and outbound network usage. CPU usage spiked to 100% and stayed that way for just over 2 minutes! That whole time the process was blocked (no dots were being printed) and there was no outbound network traffic at all. It wasn’t until after the CPU usage dropped that the data started transferring. This was clearly unacceptable.

I ran the script several more times always with the same result. I also tried uploading several different files, always with the same result. I was close to scrapping the idea of using EventMachine when I noticed that the happening library was using the “https” protocol by default. I updated my script so it specified a protocol of “http” instead.

item = Happening::S3::Item.new('my_bucket', 'my_file', :protocol => "http", ...

I ran the modified script and after a brief spike in CPU usage it dropped down to almost nothing. I could see the file data being sent immediately and dots were being printed to the terminal every second indicating that the upload was not blocking the main event loop.

So the question is, what is it doing prior to uploading the data when using SSL? I dug around in the source code for happening, em-http-request and EventMachine for a while but didn’t see anything that would explain what was happening when I tried to upload to S3 using https.

Has anyone else run into this? I put a sample project up on GitHub for people to try this out on their own machines (you need to have your own S3 account). Oh, and I was using Ruby 1.9.2 for this.

Patrick Bacon (47 Posts)

I’m a software developer at Atomic Object (since 2005) in Grand Rapids, MI.

This entry was posted in DevOps & System Admin. and tagged , . Bookmark the permalink. Both comments and trackbacks are currently closed.