The title of this post says it all. I’m super excited that I now have RSpec configured to run my tests in random order. Test interactions are gone. Assumptions have been eliminated. The test suite is improved.
In early 2012, RSpec added the option to run tests in a random order. Last week (early Agust 2012) I had a good opportunity to try it out with my current project’s test suite. Random ordering revealed three flaws in the test suite, which I’ve since eliminated:
- Our poor usage of the rspec-set library.
- Our less-than-optimal usage for Fabrication’s sequence number feature.
- The assumptions in our system tests’ navigation helpers that are not always true.
rspec-set to help test some of our modules that simply read records from the database. These test suites have lots of different cases to test, and since they’re only reading from the database, there isn’t a lot of reason to set up and tear down data between tests. It doesn’t change.
rspec-set is nice because it allows us to only build and destroy those records once. But our usage of it probably isn’t right (or it’s buggy), and we need to manually clear out records. RSpec’s new random ordering helped me find places that weren’t cleaning up after themselves well.
Fabrication has a nice feature to set up incrementing sequence numbers between fabrications. This helps ensure uniqueness across records. Do you have a field that need to be unique? Toss a sequence number into the generated value, and you don’t need to worry about uniqueness violations. We’ve gotten good at making sure we use sequence numbers, but some of our older fabricators still weren’t using them. Running the tests in a different order resulted in collisions. Now those old fabricators are using sequence numbers. Thanks again, RSpec.
Lastly, some of the navigation helpers in our system tests were always assuming the logged in user’s ID was 1; thus, they’d have code in them like
portfolio_path(1). Running the tests in a different order showed that sometimes the logged in user doesn’t always have that ID. The helpers no longer make that assumption and are now more robust.
Test interaction sucks. It is never intentional. But, of course, accidents happen. Random ordering helps find test interactions.
RSpec, thanks a bunch for running my tests in random order. I only wish I had tried it out sooner. (Thankfully, new RSpec projects have this feature turned on by default, so I can’t forget again in the future.)