Atomic Spin

Two weeks ago, I upgraded a project from rails 2.0.2 to 2.2.2. I had done some polymorphic associations, and I needed more powerful eager loading to prevent N+1 queries. The latest version of rails happens to meet my needs perfectly.

I told the customer that this work needed to be done and that it might take a few days to do it right, as it's a large project and depends on a large number of gems and plugins. The customer was very reasonable and said if it had to be done, then we'd take the time to do it right. So I did the upgrade.

I upgraded rails and tried to start the server. It wouldn't start. I followed the errors, upgrading plugin after plugin after gem until it launched. At this point, I could have deployed it to staging, crossing my fingers, hoping nothing had gone terribly wrong. Alternatively, I could've spent days exploring the application to find anything that broke. After doing that, I could have deployed to staging, crossing my fingers, hoping nothing had gone terribly wrong.

But wait, we've built our reputation on being world-class automated testing fanatics. So instead, I ran my tests. There were 1,777 spec failures and hundreds of integration test failures. I began the painstaking work of fixing tests one after another. It took 3 days, but finally the tests passed, and I was able to deploy with confidence.

It hurt a lot. I've never seen such a disruptive rails upgrade, including the jumps to 1.2 and 2.0. Half of my time zone code had to be rewritten. Almost every plugin was broken (but had new versions available). Uploads had to be tested differently. A new plugin now had some attributes protected from mass assignment, so I had to change my model creation code in a few places. Can you imagine the pain if I didn't have tests?

I would have fixed the obvious issues, and deployed to staging. My customer would have come back to me over and over and over with things that no longer worked, and each time, I'd have had to walk through the painful steps of reproducing the issue and then tracking it down to its source. It could have taken weeks, and even then perhaps no one would have noticed that time zones weren't working right until it was in production.

To be fair, some of the pain came from the tests themselves. Some of our testing techniques relied on rspec behavior that changed or on some intricacy in the framework. But I'll gladly take that pain if it saves bugs in staging or production.