Headless Chrome with Testem on VSTS-Hosted Agents

I previously wrote about building a Node app on VSTS Windows agents. Since then, we’ve started using headless Chrome on those agents. Here’s how.

Browser test automation has come a long way over the years. We used to accept having a browser flashing around in an interactive login session. Then we made browsers headless-_ish_ with [Xvfb]. Then we used truly headless browsers like [PhantomJS], even though they tended to be weaker than their desktop counterparts.

Last year, Firefox and Chrome added headless support, giving us the best of both worlds: the performance, stability, and feature set of mass-market browsers with the logistical convenience of command line applications.

The JavaScript ecosystem has embraced them. Ember.js, the front-end framework I’m currently working in, switched to headless Chrome by default [last year][ember_blog_215], and it recently [dropped support][ember_blog_3] for Phantom.

At a high level, I need to 1) download Chrome, and 2) make it available to my test framework.

###Get Chrome
We’re using VSTS-hosted agents, which [don’t][vsts_agent_software] yet have Chrome preinstalled. Where should we get it?

There’s a [VSTS task][chromium_installer_task] and a few [npm packages][npm_packages], but nothing seemed widely used or well-maintained. I suppose the JavaScript community doesn’t have a great need for such a package, as competing CI systems offer Chrome out-[of] [circle_chrome] -[the][travis_chrome] -[box][appveyor_chrome].

Luckily, I stumbled across a [blog post][benjaminspencer_post] by Benjamin Spencer, in which he pulls it out of [Puppeteer], a Chrome automation project maintained by Google. We’re not using Puppeteer, but as a source, it seems more likely to stick around (and get updated) than the other, lesser-used projects.

###Provide it to Testem
Testem’s approach of looking for [specific executables][testem_known_browsers] works well for most situations, but it won’t magically find Puppeteer’s chrome.exe deep inside node_modules. I experimented with adding it on the executable path, but then I decided to try Testem’s [custom launchers][testem_launchers] instead.

Here’s how to download and use headless Chrome for an Ember-CLI project on VSTS Windows agents:

  1. Add the Puppeteer package with e.g. yarn add --dev puppeteer. This puts the browser at a path like node_modules/puppeteer/.local-chromium/win64-536395/chrome-win32/chrome.exe.
  2. Wrap Chrome for easy access. We can avoid using this long (and dynamic) path with a short Node script that asks Puppeteer where to find Chrome, then runs it:
    const { executablePath } = require("puppeteer");
    const { execFileSync } = require("child_process");
    let exePath = executablePath();
    let args = process.argv.slice(2);
    execFileSync(exePath, args);

    Now we can run Chrome with e.g. node run-chrome.js google.com.

  3. Add a custom launcher to Testem’s configuration. Here’s my Testem.js:
    /* eslint-env node */
    module.exports = {
      "test_page": "tests/index.html?hidepassed",
      "report_file": "tmp/results.xml",
      "reporter": "xunit",
      "xunit_intermediate_output": true,
      "disable_watching": true,
      "launch_in_ci": [
      "launchers": {
        "LocalChrome": {
          "exe": "node",
          "args": [
          "protocol": "browser"

With Chrome in CI, our builds are faster and more reliable, and we’re able to raise our [target language level][ember_build_targets], which had been held back for PhantomJS. VSTS may [soon][github_issue] offer Chrome out-of-the-box, but with a little effort, we can enjoy it today.

[puppeteer]: https://github.com/GoogleChrome/puppeteer
[vsts_agent_software]: https://docs.microsoft.com/en-us/vsts/build-release/concepts/agents/hosted#software
[github_issue]: https://github.com/Microsoft/vsts-image-generation/issues/47
[ember_build_targets]: https://guides.emberjs.com/v3.0.0/configuring-ember/build-targets/
[benjaminspencer_post]: https://benjaminspencer.me/post/14/headless-chrome-vsts
[headless_chrome]: https://developers.google.com/web/updates/2017/04/headless-chrome

[travis_chrome]: https://docs.travis-ci.com/user/chrome
[circle_chrome]: https://circleci.com/docs/2.0/circleci-images/#language-image-variants
[appveyor_chrome]: https://www.appveyor.com/docs/build-environment/#web-browsers

[ember_blog_215]: https://www.emberjs.com/blog/2017/09/01/ember-2-15-released.html#toc_chrome-by-default
[ember_blog_3]: https://www.emberjs.com/blog/2018/02/14/ember-3-0-released.html#toc_browser-support-in-3-0

[chromium_installer_task]: https://marketplace.visualstudio.com/items?itemName=schlumberger.chromium-build-tasks-Preview

[puppeteer_chromium_updates]: https://github.com/GoogleChrome/puppeteer/commits/master/package.json

[testem_known_browsers]: https://github.com/testem/testem/blob/master/lib/utils/known-browsers.js

[testem_launchers]: https://github.com/testem/testem#launchers

[Xvfb]: https://en.wikipedia.org/wiki/Xvfb
[travis_xvfb]: https://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-a-GUI
[phantomjs]: http://phantomjs.org/
[npm_packages]: https://www.npmjs.com/search?q=chrome&page=1&ranking=maintenance

  • Jared says:

    Thanks John, worked like a charm. Just as a side note, I found that trying to run ember test as a task in npm would fail in VSTS. I haven’t looked to much into why this is, but I had much more luck using yarn.

    • John Ruble John Ruble says:

      I’m glad it was useful! We’re using yarn, too.

  • Steve Elberger says:

    In case anyone else had trouble with the run-chrome.js file blowing up because executablePath is an instance method and requires an instance as context:

    const puppeteer = require(‘puppeteer’);
    const { execFileSync } = require(‘child_process’);

    const exePath = puppeteer.executablePath();
    const args = process.argv.slice(2);
    execFileSync(exePath, args);

    • John Ruble John Ruble says:

      Good call, Steve. The original version of run-chrome.js in the post broke with a puppeteer update some time ago, and your version is how we fixed ours, too.

  • Comments are closed.