Learning from Pokémon GO: The Importance of Testing

Automated testing is one of the most powerful tools available to a software developer, but some people are reluctant to embrace it. With the release of Pokémon GO and its bugs seen in the first several weeks, this is a good time to discuss the importance of releasing quality code into production.

One of the  most important factors of bug-free code is a comprehensive automated test suite that covers a wide variety of potential problems.

The Background of Pokémon GO

If you’ve been living under a rock for the past few weeks and have somehow missed the global phenomenon, Pokémon GO is a popular game released by Niantic Labs earlier this month.

If you have heard about it, then you’ve probably seen many news articles talking about the huge success of the game. These articles throw around statistics about the number of downloads, the number of users, the number of minutes played, the amount of money Niantic is earning per minute–the list goes on. This is the kind of success that almost every software company dreams of. Making millions of dollars with a brand new product can’t be a bad thing, can it?

Conceptually, Pokémon GO isn’t groundbreaking. Niantic Labs developed a similar game called Ingress. However, that never saw the same amount of success that Pokémon GO did. Earlier this year, Niantic reported that Ingress had a total of 14 million downloads over the course of three years. Pokémon GO, in comparison, had that many downloads in about a week, and it’s estimated at about 75 million downloads right now.

I don’t know how much of the game code came from Ingress, but all of the location information used in Pokémon GO came straight from the Ingress databases. One way or another, Pokémon GO wasn’t built from scratch, and with $30 million in financing, Niantic should have had the resources to properly prepare for the release.

Downsides of a Rushed Release

The tradeoff to releasing Pokémon GO as quickly as possible was that there would be many initial problems. Thousands of users took to social media and to the app store to voice their complaints. The app was buggy, the servers were not responding, and Niantic forgot to mention that they received complete access to all Google accounts used in the sign-in process. There are guides online that explain how to play the game, many of which include steps on how to fix it when it crashes (hint: try force quitting!).

I am not criticizing Niantic. I am personally a huge fan of the game, but there’s no doubt that the game had—and continues to have—many problems.

It isn’t necessarily a good thing for a product to gain so much popularity in such a short period of time. A slow release means more production testing, with fewer users affected by the initial problems. Not only do the developers at Niantic have their hands full with fixing bugs, but they also have millions of players requesting many new features. Naturally, fixing critical bugs takes priority over adding new features, so those players will need to wait even longer.

It’s hard to say if Niantic properly prepared for the release of Pokémon GO by thoroughly testing the application beforehand. On their blog, they frequently reference manual testing, but there is no indication of writing any automated tests. They couldn’t have expected this kind of support for the game right away, but that doesn’t mean they shouldn’t have prepared for it. Whether or not Niantic used any automated tests while developing Pokémon GO, they provided a good example of why testing is so important.

Testing Is Always Worth It

Not everybody agrees on the idea of automated tests. Some people say things like, “I don’t need a test to show that my code works.” This is true to some extent, and I agree that not everything needs testing. However, the larger the codebase gets, and the more complicated the logic becomes, the harder it is to prove that the code actually works.

This is where different kinds of testing come in: unit, integration, functional, acceptance, and system tests, to name a few. Each of these testing methodologies is different yet complementary. For example, unit tests will focus on specific methods of a class, while acceptance tests will test the implementation of whole features.

I was recently talking with a friend who works for a company that doesn’t believe in the importance of testing. He told me he felt nervous making changes in the codebase he’s working on, and he said something along the lines of, “As soon as I make any changes to the code, I can see if my feature works, but I have no way of knowing if I broke anything else.” This conversation made me realize how much I appreciate the teams I work with, because I never have to wonder how the changes I make will affect the rest of the codebase.

Preparing for Release

Although some of the things that went wrong with Pokémon GO were out of Niantic’s control (for example, the overloaded servers during the first few days), they still could have prepared for the worst.

The majority of the game is developed with the Unity engine, which has a good testing library called Unity Test Tools that supports unit and integration tests. The rest of the application (signing up/signing in, network connectivity, etc.) is native Android and iOS code, and it supports a wider variety of testing frameworks.

From the game engine to the network and to the future Bluetooth integration, all areas of this game could be tested. Unit testing is a good way to show that individual objects work well by themselves, but integration testing is necessary to ensure that they work well with one another.

Even though Niantic did not know how their servers would function, they could have simulated a bad network connection within a testing suite. An acceptance test would have been a great place to see how the application would handle poor network connectivity. Knowing how the application reacts to bugs is the first step to fixing and preventing them.

There are many other things that should be tested in this application. A few examples include:

• Account creation/logging in
• Connecting to the network
• Correctly identifying user location
• Loading persisted data, such as login information
• In-app purchases
• Basic game functionality, such as:
  • Catching a Pokémon
  • Gaining experience and leveling up
  • Using items
  • Fighting battles at a gym

Good tests show both successes and failures of the feature being tested, when applicable. For example, with an application that relies on network connection, there should be cases for when the application can and cannot connect, and both should be handled appropriately.

Conclusion

On one hand, writing a comprehensive test suite, especially for larger applications, takes a lot of time. Yet, it’s not a waste of time. Not only does it ensure that the application functions the way it’s expected to, but it also helps with fixing bugs when they inevitably arise.

Applications that incorporate behavior-driven development techniques (especially English-like tests) will shorten the learning time when a new developer joins a project or when support is required in the future. In the long run, they will save time and effort, especially during the support period, when active development slows down.

If you are already in the habit of testing your code, you can use Pokémon GO as a learning opportunity to look for edge cases to test (however unlikely they seem).

If you don’t use automated tests in your codebase, I encourage you to prepare your applications for release through a good test suite. Do you want to spend all your time after a big release quickly fixing bugs, or would you rather be confident that everything will work before your code is ever seen in production?