Testing Asynchronous Behavior in JavaScript with Selenium

Full stack browser testing of web applications is awesome. It validates that your application works end-to-end and allows you to check actual user workflows. For the last year and a half, I’ve been using Selenium to test a JavaScript web application we’re developing using Backbone.js.

Unfortunately, web apps that use JavaScript a lot can be challenging to test with Selenium.

Selenium was designed for testing traditional websites and isn’t well equipped out of the box to handle asynchronous behavior in JavaScript apps that might change the page. Even something simple like looking for an element that’s put on the page in response to an AJAX request will fail with Selenium because it has no idea about the internals of your web app. Selenium just knows that the content it’s looking for is not on the page right now.

A Flawed Workaround

The poor man’s solution to this is just to add lots of sleeps into your test code, so that the test waits long enough for any asynchronous behavior to complete. There are various libraries on top of Selenium that will abstract away sleeping into nicer APIs, but they all share the same core disadvantage. Specifically, they all require you (the programmer) to reason about and provide details about your app’s inner working to your tests, because the framework isn’t capable of figuring them out.

In a JavaScript application of any significant complexity, this sort of behavior starts to become impossible to reason about. As a result, you end up with buggy, unreliable tests. Ember and Angular have recognized this and responded by releasing end-to-end testing solutions that have knowledge of the framework internals.

Unfortunately, for anyone doing JavaScript development without those frameworks, these problems still exist. In the application my team has been working on, these problems had been building over the course of the project, and led to frequent frustrations with our acceptance test suite. Some developers had even moved entirely away from using the suite because they didn’t trust it.

Engineering a Real Solution

With all the problems in our test suite before us, our team decided to devote some serious time into finding a solution for writing Selenium tests.

We had a couple of non-starter attempts that tried to communicate specific parts of our application’s state to our Selenium server, but those quickly grew unmaintainable and polluted our application code with testing concerns. We wanted something that we could write once, that would sit transparently on top of our application so we never had to think about it again.

Then, my teammate Ken had an “Aha!” moment. Since the browser (and therefore Selenium) already blocks on synchronous code being executed, and there are only a few well-defined ways to run asynchronous code, we just needed to globally hook into all sources of asynchronous code and wait for those to finish. (It was one of those head bang moments, where you feel pretty dumb for having tried any other way.)

When we thought about it more, there were really only three sources of asynchronous code we needed to handle: AJAX calls, and calls to setTimeout or setInterval. Since we were using jQuery, we were already able to monitor AJAX calls using the jQuery.active property.

setTimeout and setInterval are a little trickier. There is no way to just get a list of currently active timeouts or intervals from the browser, so we actually need to override the implementation to do some monitoring. Fortunately, this is pretty easy, and Mattie and I were able to whip something up in about an hour.

The code looked something like this (we used underscore.js to make some parts more readable):

// Set up container to hold timeout and interval state
window.testing = {
  timeouts: {},
  intervals: []
};

// Make a copy of the original setTimeout function
window._setTimeout = window.setTimeout;
window.setTimeout = function(callback, timeout) {
  // We need a handle to store our timeout under, we can't just use the
  // timeout ID because we don't know it until after we create the timeout
  var handle = _.uniqueId();

  // Call the old setTimeout function
  var timeoutId = window._setTimeout(
    function() {
      // The callback is the function we were originally deferring
      callback();

      // Once a timeout completes, we need to remove our reference to it
      delete window.testing.timeouts[handle];
    },
    timeout
  );

  // Store the id of the timeout we just created so it can be queried
  window.testing.timeouts[handle] = timeoutId;
  return timeoutId;
};

// Make a copy of the original clearTimeout function
window._clearTimeout = window.clearTimeout;
window.clearTimeout = function(timeoutId) {
  // Call the original clearTimeout function to actually clear the timeout
  var returnValue = window._clearTimeout(timeoutId);

  var timeoutToClear;
  // Look over all the timeouts we have stored and find the one with
  // the timeoutID we just passed in
  _.each(window.testing.timeouts, function(storedTimeoutID, handle) {
    if(storedTimeoutID === timeoutId) {
      timeoutToClear = handle;
    }
  });

  // Delete our stored reference to the timeout
  delete window.testing.timeouts[timeoutToClear];

  return returnValue;
};

// Make a copy of the original setInterval function
window._setInterval = window.setInterval;
window.setInterval = function(cb, interval) {
  // Use the original setInterval function to schedule our interval
  var intervalId = window._setInterval(cb, interval);
  // Store the ID returned by the setInterval call
  window.testing.intervals.push(intervalId);
  return intervalId;
};

// Make a copy of the original clearInterval function
window._clearInterval = window.clearInterval;
window.clearInterval = function(intervalId) {
  // Use the original clearInterval function to clear our interval
  var returnValue = window._clearInterval(intervalId);
  // remove the passed interval ID from our list of IDs
  window.testing.intervals = _.without(window.testing.intervals, intervalId);
  return returnValue;
};

Then, from Selenium, we just need to write a JavaScript snippet that checks the state of the three properties we are interested in, something like:

jQuery.active === 0 
&& Object.keys(window.testing.timeouts).length === 0 
&& window.testing.intervals.length === 0;

The important thing is, if all three are clear, it’s impossible for the browser to run more code or change its state until we perform some UI action. Therefore the current state of the application is safe to query for our test.

Results

This technique worked great for us. We no longer had tests erroring out inconsistently because of async behavior running at different times, and we had to put less thought into constructing our tests because everything just worked.

If you are having trouble controlling asynchronous behavior in your automated browser tests, I recommend giving this a shot. Let me know in the comments if you have any success with it or have any different solutions.