Even after 10 years, I find I occasionally need a reminder to check all the obvious things when something goes wrong.
I just spent about a day debugging an issue with tests hanging, but only on one machine under continuous integration. The other machines were all fine. I assumed one of my tests was just broken: maybe I was polling for an update in the UI forever or something like that.
I definitely took the long way around trying to debug things. I analyzed which tests were running and how far each was getting by adding print statements that I could view in the CI output. I searched out all the tests that triggered events, just in case my mock event handler was doing Something Strange.
Eventually I started opening up rubygems and annotating the third party code inside them. I got closer to finding the specific point where everything went haywire, but I wasn’t really getting any closer to actually *solving* the problem.
I also swore a lot. I think it helped, a little.
Eventually I traced down the rabbit hole further and found where a third party gem was shelling out to third party executable, and that’s where I found out I had a version mismatch: in CI, the software was something like 4 major revisions behind what I had on my laptop.
Despite triple checking that the versions matched, I missed it. The reason that happened is because I was looking at my Gemfile and I saw that I had the version pinned, so it couldn’t be that. But it turned out that it was a dependency of a dependency, so it wasn’t in the Gemfile. In fact, it wasn’t even in the Gemfile.lock because it was a system executable and wasn’t part of the bundle. I’ve been spoiled by rbenv and Bundler for so long that I assumed they were working as I intended.
This was a good reminder to make sure we step back and look at the big picture. If I’d stopped looking at individual lines of code and instead made sure to identify the complete stack of components to compare, I’d have found this issue very quickly. Next time you see something inexplicable happen, try to take in the whole problem first before you get lost in the weeds.