Parallelizing Ember Tests Across CI Workers

One of CircleCIʼs killer features is automatic test parallelization: Circle can dramatically improve your build times by divvying up your tests across multiple _build containers_. Split three ways, this brings our 55-minute build time down to about 23 minutes:

Those three large bars represent our [automatically-balanced](https://circleci.com/blog/announcing-automatic-test-balancing/) RSpec test suite. See that lone bar on the right side, keeping container #0 busy while #1 and #2 take a break? Those are our [Ember tests](https://guides.emberjs.com/v1.13.0/testing/). Circle is unable to automatically split them, but we can do it [manually](https://circleci.com/docs/2.0/parallelism-faster-jobs/)! Here’s how.

## Approaches
My first choice would be to use the same file path wildcard approach that we use for RSpec. We would specify e.g. `”tests/**/*.coffee”`, and Circle would call our command with a subset of these files on each worker. Then, if we could coax the output into one of Circle’s supported [metadata formats](https://circleci.com/docs/test-metadata/), we could even take advantage of automatic rebalancing. Unfortunately, Ember-CLI [doesn’t seem](https://github.com/ember-cli/ember-cli/blob/master/lib/commands/test.js) to be able run individual tests by file.

Instead, we can use a couple of environment variables– `CIRCLE_NODE_INDEX` and `CIRCLE_NODE_TOTAL`–and come up with the commands to run ourselves.

So here’s the meat of the problem: Given _n_ workers, how can we run approximately _1/n_ of our Ember tests?

## Solution
First, we need to make `NODE_INDEX` and `NODE_TOTAL` information available to the test runner. Looking through Ember-CLI’s command line parameters, this one caught my eye:

`–query (String) A query string to append to the test page URL.`

This string, given to Ember-CLI, will be tacked onto the end of the address of the QUnit test page; parsed parameters will be accessible in JavaScript as `QUnit.urlParams`.

The next step is to control which tests are run. We can do this by overriding the [TestLoader](https://github.com/ember-cli/ember-cli-test-loader)ʼs `shouldLoadModule` function.

The final piece of the puzzle is, given the node number information and the _name_ of a test, how should we decide whether to run it? Here’s what I came up with:

// place this in a script element just after test-loader.js in tests/index.html
jQuery(document).ready(function() {
  var TestLoader = require('ember-cli/test-loader')['default'];
  TestLoader.prototype.shouldLoadModule = function(moduleName) {
    if (QUnit.urlParams.workerIndex && QUnit.urlParams.numWorkers) {
      var n = moduleName.split('')
                .map(function(c){return c.charCodeAt(0);})
                .reduce(function(a,b){return(a+b)});

      return (n % parseInt(QUnit.urlParams.numWorkers) == parseInt(QUnit.urlParams.workerIndex));
    }
    return moduleName.match(/\/.*[-_]test$/);
  };
});

With this in place, we can run one share of the tests like this:

`./node_modules/.bin/ember test –query workerIndex=1\\&numWorkers=3`

I dropped this into a rake task:

desc "Run Ember tests split across CircleCI containers"
task :ember_circleci do
  raise "missing CIRCLE_NODE_INDEX env var" unless ENV["CIRCLE_NODE_INDEX"].present?
  raise "missing CIRCLE_NODE_TOTAL env var" unless ENV["CIRCLE_NODE_TOTAL"].present?
  Dir.chdir "web-client" do
    sh "node_modules/.bin/ember test --query workerIndex=#{ENV['CIRCLE_NODE_INDEX']}\\&numWorkers=#{ENV['CIRCLE_NODE_TOTAL']}"
  end
end

And then finally invoked it as a custom test command in Circle’s `circle.yml` configuration file like this:

test:
  override:
    - bundle exec rake spec:ember_circleci:
       parallel: true

##Results
Splitting our Ember tests three ways saves about two minutes of build time. It’s a small optimization, but for a process we use many times per day, it’s well worth the effort.

Because they are divided dynamically, though, we can get even faster builds by cranking the parallelism up higher:

The speedup is great, but this approach is not ideal: It required a fair amount of custom setup, and it may need maintenance as third-party components change over time. Additionally, because of limits in the test tooling, we can’t take advantage of Circle’s automatic balancing.

I’m reminded of a quote from an EmberConf 2016

This slideshow could not be started. Try refreshing the page or viewing it in another browser.

(https://twitter.com/katiegengler/status/715284618128175104) about Ember Addons: _”Someone must have done this before, and I bet they did it better than I would.”_ I’m optimistic that in the future, better tooling for parallel testing could be provided by an addon for Ember-CLI.