Article summary
Programming is fun. Really, it is. But occasionally, it will test you more than you’re testing it. There are those times when your patience might wear thin, and you find yourself questioning your own sanity. This is a story about one of those times.
Fear and Loathing in Acceptance Tests
It was 3:30 pm on a Friday afternoon. The day was going well. We were working on a relatively straightforward feature: a user login. Before we began to implement the feature, we had set up an acceptance test to simulate a user logging in, and we saw the test fail as expected—some classic test-driven development.
After some solid development time, we were able to log in locally without issues. Unfortunately, when we ran the test again, it was still failing. “Darn, we must have missed something,” we thought. Maybe there was an issue with the test setting up a user record in the database. Perhaps there was just another fluke that let us log in locally. So, we began debugging.
As we went along debugging our test, we struggled to find an obvious issue with our code. With a little more digging, we found that the server wasn’t receiving the expected login information. The test user we had created was pretty simple. Username: billy. Password: abc123. But in the test, the password was coming in as abc12.
Okay, we thought, maybe the “Submit” button was being clicked too quickly after the password input was filled in, and we should make sure there was a helper in place to verify that an input had the proper value before moving on to the next step. So, we added the helper, but then…the helper failed when trying to find the proper value. It still only got to abc12. Weird.
Well, maybe, for whatever reason, the last character wasn’t being sent, even though it was working for the username. We tried having the test type in abc1234. We ran it, and we saw: abc124. Commence the perplexed googling.
Our acceptance tests were using WebDriver, so a lot of our searches looked like, “Keys not being sent in WebDriver test.” We seemed to find some results suggesting that the locale setting on our browser might be incorrect. So, we tried explicitly setting the locale to en-us. Still, we weren’t seeing any 3s.
After even more digging, we decided that for the sake of budgeting our time on the project, we would just write a helper to replace any 3 we’re sending with the Numpad 3 key, which worked just fine. We got the test passing and continued on.
One Week Later…
Shortly after that encounter, all of our acceptance tests suddenly started failing. While running them on my machine to try and figure out what was wrong, I noticed some very peculiar behavior. As the tests were running in Chrome, the settings tab was launching and stealing focus from the original tab containing our application. This caused the tests to fail. Obviously, that wasn’t good.
Once again, I went to consult my trusty search engine friend. The first solution I found was to upgrade the version of Chromedriver we were using from 2.30 to 2.33. Sure enough, after upgrading, the problem went away. Then we had an idea. “I wonder if the test can actually send the 3 key now?”
We took out the little helper we had for replacing the 3s, ran our login test, and what do you know, it worked! After that, one of my teammates decided to go check out the ChromeDriver version change logs. As he looked through the changes, he began to chuckle and called me over. I glanced at the list he was pointing to, for the 2.32 version, and sure enough, there it was: “Fixes a bug where Chromedriver fails to send characters 3 and # keys on Mac Chrome 62+.” Sigh.
Moral of the Story
On top of solving the actual problem with our tests, this experience was a good teachable moment for me. First, I learned that checking the version of your software and the change log can lead to some very important discoveries. It might even lead you to the solution you need right away. Second, it’s another testament to the importance of comprehensive software testing and covering as many use cases and configurations as you possibly can. And third, it’s a reminder that software is still written by humans, and any of us can make mistakes.
If you have any other entertaining tales of programming adventures, feel free to share in the comments. I’d love to read them!