Parallelizing Ember Tests Across CI Workers

One of CircleCIʼs killer features is automatic test parallelization: Circle can dramatically improve your build times by divvying up your tests across multiple build containers. Split three ways, this brings our 55-minute build time down to about 23 minutes:

Those three large bars represent our automatically-balanced RSpec test suite. See that lone bar on the right side, keeping container #0 busy while #1 and #2 take a break? Those are our Ember tests. Circle is unable to automatically split them, but we can do it manually! Here’s how.

Approaches

My first choice would be to use the same file path wildcard approach that we use for RSpec. We would specify e.g. "tests/**/*.coffee", and Circle would call our command with a subset of these files on each worker. Then, if we could coax the output into one of Circle’s supported metadata formats, we could even take advantage of automatic rebalancing. Unfortunately, Ember-CLI doesn’t seem to be able run individual tests by file.

Instead, we can use a couple of environment variables– CIRCLE_NODE_INDEX and CIRCLE_NODE_TOTAL–and come up with the commands to run ourselves.

So here’s the meat of the problem: Given n workers, how can we run approximately 1/n of our Ember tests?

Solution

First, we need to make NODE_INDEX and NODE_TOTAL information available to the test runner. Looking through Ember-CLI’s command line parameters, this one caught my eye:

--query (String) A query string to append to the test page URL.

This string, given to Ember-CLI, will be tacked onto the end of the address of the QUnit test page; parsed parameters will be accessible in JavaScript as QUnit.urlParams.

The next step is to control which tests are run. We can do this by overriding the TestLoaderʼs shouldLoadModule function.

The final piece of the puzzle is, given the node number information and the name of a test, how should we decide whether to run it? Here’s what I came up with:

// place this in a script element just after test-loader.js in tests/index.html
jQuery(document).ready(function() {
  var TestLoader = require('ember-cli/test-loader')['default'];
  TestLoader.prototype.shouldLoadModule = function(moduleName) {
    if (QUnit.urlParams.workerIndex && QUnit.urlParams.numWorkers) {
      var n = moduleName.split('')
                .map(function(c){return c.charCodeAt(0);})
                .reduce(function(a,b){return(a+b)});

      return (n % parseInt(QUnit.urlParams.numWorkers) == parseInt(QUnit.urlParams.workerIndex));
    }
    return moduleName.match(/\/.*[-_]test$/);
  };
});

With this in place, we can run one share of the tests like this:

./node_modules/.bin/ember test --query workerIndex=1\\&numWorkers=3

I dropped this into a rake task:

desc "Run Ember tests split across CircleCI containers"
task :ember_circleci do
  raise "missing CIRCLE_NODE_INDEX env var" unless ENV["CIRCLE_NODE_INDEX"].present?
  raise "missing CIRCLE_NODE_TOTAL env var" unless ENV["CIRCLE_NODE_TOTAL"].present?
  Dir.chdir "web-client" do
    sh "node_modules/.bin/ember test --query workerIndex=#{ENV['CIRCLE_NODE_INDEX']}\\&numWorkers=#{ENV['CIRCLE_NODE_TOTAL']}"
  end
end

And then finally invoked it as a custom test command in Circle’s circle.yml configuration file like this:

test:
  override:
    - bundle exec rake spec:ember_circleci:
       parallel: true

Results

Splitting our Ember tests three ways saves about two minutes of build time. It’s a small optimization, but for a process we use many times per day, it’s well worth the effort.

Because they are divided dynamically, though, we can get even faster builds by cranking the parallelism up higher:

The speedup is great, but this approach is not ideal: It required a fair amount of custom setup, and it may need maintenance as third-party components change over time. Additionally, because of limits in the test tooling, we can’t take advantage of Circle’s automatic balancing.

I’m reminded of a quote from an EmberConf 2016 presentation about Ember Addons: “Someone must have done this before, and I bet they did it better than I would.” I’m optimistic that in the future, better tooling for parallel testing could be provided by an addon for Ember-CLI.