Testing & Tooling in EmberJS

Like many other Atoms, I’ve recently been doing some work with EmberJS. Ember is an awesome web development toolkit with some really killer features. One of my favorite parts of Ember is how easy it is to test. The framework comes bundled with a system testing framework, and its object model makes unit testing a breeze. Combine all that with a great test runner that has CI integration, and you have a really awesome testing ecosystem for your new app.

System Testing with Ember Testing and QUnit

EmberJS comes bundles with a great system testing framework called ember-testing. This framework allows you to make high-level assertions regarding your applications’ state, mostly by querying and interacting with the DOM through JQuery in a black box fashion. Since the tests are pure JavaScript however, we also get direct access to our running application if needed for testing.

The Basics

The framework comes bundles with a few simple helpers to do common operations such as clicking a button, filling in a text field, or querying for the presence of a certain DOM element. A typical system test using ember-testing might look like this:

module 'Sign in tests',
  setup: ->
    App.reset()

test 'Signing in successfully goes to the main page', ->
  visit('/signIn')
  fillIn('.app-username-field', 'username')
  fillIn('.app-password-field', 'password')
  click('.app-sign-in-button')
  andThen ->
    ok findWithAssert('.app-main-content')

Notice how the above code is just a set of synchronous function calls? That’s because ember-testing abstracts away all the details of waiting for any asynchronous actions caused by one of your function calls. For example, the step that clicks the sign-in button would likely have all sorts of asynchronous behavior, such as:

1. Handling the initial click action.
2. Sending a server request to authenticate the entered credentials.
3. Receiving a response from the auth request, and saving some sort of session token to the client.
4. Transitioning to another route in the application (which would likely load more data from a server).

Thats a lot of things that need to happen before we can run our assertion checking for the main page content to appear. Fortunately, ember-testing knows that your app will have to perform lots of async actions and will wait for things like router transitions and pending ajax requests to finish before running the next step in your test. There is even a feature currently being tested that will allow you to define addition conditions that will cause your app to pause between steps.

Extensibility

Ember-testing is also easy to extend in order to add useful assertions and DRY up your test code. Writing a helper is as simple as:

Ember.Test.registerHelper 'shouldHaveElementWithCount', (app, selector, n, context) ->
  el = find(selector, context)
  count = el.length

  unless n == count
    throw new Ember.Error "Expected to find selector #{selector} '#{n}' times but found it '#{count}' times"

  el

Here we define a simple helper that checks that a DOM element is present exactly as many times as we expect it to be. Since this is just an assertion method, we don’t need to do anything with async behavior. If we did need a helper that’s concerned about async side-effects, it’s just as easy to write:

Ember.Test.registerAsyncHelper 'clickFirst', (app, selector, context) ->
  $el = findWithAssert(selector, context).first()
  $el.click()

The only difference is that we define our helper with registerAsyncHelper instead of registerHelper.

The Ugly Part

Despite all its great features, ember-testing isn’t all rainbows and butterflies. One issue I had while using ember-testing was running into the following error:

Assertion Failed: You have turned on testing mode, which disabled the run-loop’s autorun. You will need to wrap any code with asynchronous side-effects in an Ember.run.

This error is annoying because:

1. It gives you no context as to what code in your application caused the error.
2. The error message itself is pretty meaningless unless you understand how the Ember runloop works.

Taras Mankovski does a good job explaining the fine points of what this error means here, but the short story is that in order to avoid this error, you need to carefully wrap any asynchronous code in an Ember.run callback.

For example the following code would cause an error in Ember testing:

p = $.ajax('some/url')
p.then (data) =>
  @someobject.setProperties data

To fix it you would need to wrap the promise resolution in an Ember.run block like so:

p = $.ajax('some/url')
p.then (data) =>
  Ember.run =>
    @someobject.setProperties data

While I understand the reasoning behind this, I despise the fact that I have to bloat my code with a bunch of additional callbacks that are only there to make my test cases pass. In my particular situation, all of these errors were coming from jquery ajax requests that I was firing off (the app I am working on does not use Ember data). So after trying a few alternatives, I ended up making a small change to my copy of jquery.js so that ajax promise resolution is wrapped in an Ember.run callback. While normally making a change to such a massive vendor library would be a bad idea, in my case it was a very isolated change, and I only used the modified jquery when running my system tests.

Unit Testing with Mocha, Chai, and Sinon

For unit testing any JavaScript application, my personal preference is to use Mocha, Chai, and Sinon. These tree libraries together create a great testing environment with a solid spec style structure, a comprehensive set of assertions, and a great mocking/stubbing library.

Ember itself makes unit testing very easy with its object model, which simplifies instantiating components in isolation to test. Let’s look at a simple example controller and see how easy it is to unit test with this stack:

App.TestController = Ember.ObjectController.extend

  fieldLength: (->
    @get('fieldData').length
  ).property 'fieldData'

  actions:
    save: ->
      @get('store').saveWidget(@get('fieldData'))

module.exports = App.TestController

Here we have a simple controller with a computed property and an action. Testing this controller is very easy:

describe 'Test Controller', ->
  before ->
    @DescribedClass = require 'controllers/TestController'
    @sandbox = sinon.sandbox.create()

  beforeEach ->
    @sandbox.restore()
    @subject = @DescribedClass.create model: Ember.Object.create()

  describe 'Computed Properties', ->
    describe '#fieldLength', ->
      it 'is the length of the fieldData property', ->
        Ember.run =>
          @subject.set 'fieldData', 'abc123'

        expect(@subject.get 'fieldLength').to.equal(6)

  describe 'actions', ->
    describe '#save', ->
      it 'uses the store to save a widget with the current data', ->
        store = require('lib/tore').create()
        fakeData = 'abc123'
        spy = @sandbox.stub(store, 'saveWidget').withArgs(fakeData)

        Ember.run =>
          @subject.setProperties
            store: store
            fieldData: fakeData

        @subject.send 'save'

        expect(spy).to.have.been.calledOnce

Ember makes it easy to instantiate objects and set their properties as needed, which greatly simplifies unit testing.

Testing Handlebars Helpers

Unit testing handlebars helpers can be a little trickier, as it is not immediately obvious how to access the helper function directly in a unit test. Fortunately, I was able to figure out how to access these helpers pretty quickly by inspecting the Ember.Handlebars object in Chrome. I wrote the following helper function that I use in my unit tests for handlebars helpers:

  withHandlebarsHelper = (helperName) ->
    Ember.Handlebars.helpers[helperName]._rawFunction

This helper returns a callable version of your handlebars helper that you can unit test.

Making it all run smoothly with Grunt, Karma, and TeamCity

Building and Testing with Grunt and Karma

A great testing framework is only as good as the tooling that supports it. As I’ve mentioned in the past, my preferred toolkit for any pure JavaScript project is Grunt. Grunt integrates nicely with my favorite JavaScript test runner, karma. Karma has some nice features, including supporting different testing libraries and running tests in multiple browsers. Karma also has a large number of plugins to extend its functionality.

Continuous Integration

CI is very important to us at Atomic, so I was very happy to learn that my Grunt/Karma setup was very easy to integrate with our CI solution, TeamCity.

To start with, this stack is very light on server dependancies. The only requirements for our build machine are Node.js and a browser. Karma also has some nice plugins that help us get some more details about our CI builds beyond whether they passed or failed.

First off, Karma has a handy JUnit XML reporter. JUnit XML is a standard format for reporting test results, and TeamCity can be configured to read the generated reports. This gives us useful feedback like which tests failed and why, rather than just a failed build message.

Karma also has a coverage reporter plugin that can be configured to output coverage statistics that TeamCity can read and display.

These two features together transform our CI output from this:

to this:

Final Thoughts

EmberJS has been a lot of fun to work with, and I have found it to be the easiest to test JavaScript framework I’ve encountered. What have been your experiences with testing EmberJS applications?