Testing HTML5 Offline Features Against a Remote Server Using Capybara/Selenium

We use a wide variety of technology stacks here at Atomic, so invariably newer Atoms like me are assigned to projects where all or almost all of the technology is new to us. Since I’ve joined Atomic, I’ve gotten fairly familiar with the LAMP (Linux / Apache / MySQL / PHP) stack. (Except for the PHP; when it’s up to us, a lot of Atoms have a preference for Ruby.)

However, lately I’ve been working on a solo project using what is — to me — quite foreign: Windows / IIS / MS SQL / PHP. It’s a unique technology stack that forms its own delightful acronym. One of the main reflections I’ve had as a result of working in this different environment is the importance of making sure your tests mirror the customer’s production environment.

Environmental Differences between Development and Production

The project I’m working on was written some time ago, and (for a number of reasons) the original developer elected to run the integration and acceptance tests in the MAMP technology stack, which is basically LAMP except on Mac OS X. That’s quite different from the customer’s environment, and the syntax between MS SQL and MySQL varies widely. That said, PHP’s PDO library wraps around the differences between MySQL and MS SQL fairly well, so using MySQL for the tests is not quite as scary as it sounds.

Nonetheless, I was concerned that there would still be subtle differences in how the code behaves between our development environment and the customer’s production environment — a source of frustrations and panic attacks for a great number of programmers. So I set out to mirror the strategy we used for another app we wrote using the WIMP technology stack:

1. Develop locally on the Mac.
2. Deploy to a test Windows server.
3. Test changes remotely.

My decision to change the testing environment offered no shortage of challenges, but I want to focus particularly on one of the most nefarious ones: how to test the application’s behavior when it loses its internet connection.

Testing a Lost Connection to a Remote Server

The application uses HTML5 features to store an application cache and uses local storage to maintain encrypted login information and to track user input while disconnected. When the app was tested locally under MAMP, testing what happens when the client loses its connection to the server was as simple as sudo apachectl stop. Not so once the server was moved to another machine, on a markedly different OS.

Attempts to Mimic Disconnecting the Client From the Server

Originally, I intended to send the server a message to tell it to stop. However, after a lot of digging, I abandoned this idea. IIS is justifiably resistant to the idea of shutting down and restarting an app from the outside. And while I could install and use ssh to access the console and manipulate the app that way, the situation we are testing is actually a connectivity issue on the client. In the end, I really wanted to mirror the production environment as closely as possible.

I brainstormed for a while and eventually found a viable — if clunky — solution: just turning off my Airport (sudo ifconfig en1 down). It’s guaranteed to mimic the desired conditions, which makes it a reliable test. Unfortunately, running tests that way was very disruptive to my workflow, and pretty slow to boot — every time a test wants to reconnect, it has to wait for the Airport to find the network.

A Better Solution: Selective Packet Filtering

My current solution is a lot more targeted. I am using pf.conf and pfctl to drop all packets coming from the test server.

Here are additions to /etc/pf.conf:

table persist file "/etc/pf.blocked.ip.conf"
ext_if="en1"
block drop in log (all) quick on $ext_if from to any

This creates a table called blockedips (stored in file /etc/pf.blocked.ip.conf) and indicates that any packets from an IP address in that table should be dropped. Next, before running the tests, I need to make sure that pfctl is active and using the pf.conf file:

sudo pfctl -f /etc/pf.conf
sudo pfctl -e

Now I can toggle whether or not packets are dropped for a web address.

Blocking a site:

sudo pfctl -t blockedips -T add my.website.com

Restoring it:

sudo pfctl -t blockedips -T delete my.website.com

Unforeseen Complications with Capybara/Selenium

Blocking just one site this way is faster and less disruptive than disabling my Airport, but my exhilaration at getting it working was short-lived. Unfortunately, manipulating your packet filter in this way exposes a bug in Selenium using Firefox that I have yet to be able to work around. Specifically, when I stop blocking the server, I can see that my Selenium-driven Firefox has reconnected, but when I try to interact with the page in Selenium code (even getting the content I already see with my own two eyes) it stalls out for a minute, sometimes two.

Luckily, I was able to get around this issue by switching over to the Chrome driver for Selenium. It reconnects much more quickly than Firefox and — most importantly — never stalls out and makes my test fail erroneously.

Once you’ve downloaded the Chrome driver, here’s how to get it running (with a mobile phone user agent) in Capybara:

Capybara.register_driver :mobile_browser do |app|
  switches = ['--user-agent=iPhone'] # or something similar for user agent
  driver = Capybara::Selenium::Driver.new(app, :browser => :chrome, :switches => switches)
  # resize browser to your preferred dimensions
  driver.browser.manage.window.resize_to(480, 725)
  driver
end

Conclusion

Despite the difficulties I still think this process was the right decision. Now I can be more confident when it comes time to deploy the app. Anyone who has worked in software for over a year or two will tell you that a smooth deployment is worth its weight in gold.