Due to a project’s integration tests spuriously reporting failures with capybara-webkit on our CI servers, we switched to the capybara-selenium driver. In the process of switching, we discovered that a number of tests needed to be tweaked or partially rewritten.
Why? It turned out that capybara-webkit was allowing us to interact with invisible or hidden DOM elements, but capybara-selenium prohibited this. We realized our tests had been suffering due to being allowed this leniency.
Some of the issues stemmed from naughtiness on our part. These were tests that intentionally interacted with hidden DOM elements. One example would be where, rather than exercise the behavior of a tag editing field as if it were a real user, we were a little lazy and stuffed data directly into its underlying hidden input field. By being forced to avoid this, our tests have improved in quality because they exercise more of our app.
For other tests, we discovered there were timing issues related to animations. We make use of Twitter Bootstrap‘s modal dialogs, which have their own built-in animations. The switch to the selenium driver showed us that we were manipulating and even submitting forms in these modal dialogs before the dialogs could be shown to the user. Sometimes, there’s no material harm from this. In other cases, unexpected states were reached due to manipulating a form or interacting with buttons before necessary JavaScript callbacks had been fired.
Regardless, it was an eye-opening experience to realize that our tests were not running at all like we expected. Now that we’ve corrected these issues, we have more confidence in our test suite and run into much less inexplicable or odd problems while writing tests. If your situation allows for it, I highly recommend setting capybara to not allow interacting with hidden DOM elements.