Article summary
Testing mobile applications across multiple platforms is tricky. Software developers are now faced with an increasingly heterogeneous mix of mobile devices and operating systems. Meanwhile, people have increasingly high expectations for their user experience.
I serve as a board advisor to an Ann Arbor-based startup whose native mobile applications are a very important part of their product ecosystem. They recently asked me for technical advice on how to approach testing their mobile applications and what to watch out for.
I reached out to Atomic’s ace exploratory tester, Phil Kirkham, to get his take on a testing approach for mobile apps. Phil has experience testing hundreds of software applications (mobile and otherwise). In this post, I’ll be sharing some of his sage advice, along with some of my own thoughts on the subject.
The Hardware
When creating a testing plan, consider which platforms you’re targeting for release. Each has unique challenges.
(Are you supporting tablets? If so, add in an iPad and an Android tablet with similar considerations.)
Apple
iOS usually has fewer issues than Android due to standard models and OS versions. iOS users receive regular operating system updates, whereas Android users (those not using Pixel phones) often get stuck on the version/flavor of Android that came pre-installed on their device, as Android OEMs are not required to push new Android versions.
On iOS, be sure to test multiple screen sizes, especially small phones such as iPhone SE or 12 Mini. Testing across products (low end to high end) will not only help ensure the UI is responsive to various screen sizes, but it will help ferret out performance issues on lower-end Apple devices.
Run both the current OS and the previous one. Be aware that September is when Apple tends to release new versions. If you’re going to do a release around that time, be sure to have a phone running a Beta version.
Android
With Android, it’s especially critical to test on the old operating system versions and with a lower quality (slower) phone. If you skip this step, you set yourself up for a lot of quality complaints coming from users with older devices.
Gather market share data to work out which models and OS versions to cover. I’d recommend not supporting any version of Android with less than 5% market share. As of this writing, that’s 7.0 Nougat and earlier.
Internal & Closed Beta Testing
Developers
Developers should, of course, write automated tests during development. But they should also test new functionality on an actual device. It’s easy and convenient to test new features in a simulator, but whenever possible, developers should also manually check a phone/tablet to make sure their new work functions properly.
The Project Team
Get help from the product development team (developers, designers, testers, product managers, etc.) and internal company stakeholders. These people are close enough to the project that they are likely involved day-to-day or week-to-week. If the development team is using an agile process, then new features and general app functionality should be tested each sprint. If not, identify a release cadence aligned with internal releases to test. Most of these people will be testing using TestFlight (on iOS) or Google Play Console (on Android). Other options like HockeyApp are available and work well.
Friends & Family
As a last internal step, create a test plan and schedule mob testing sessions where everyone on the team can do some exploratory testing together. Get marketers, managers, sales people, friends, and family involved.
Beta Testers
Once the software is in a good enough place to expose it to broad external testing, tools like TestFlight can be used to invite large groups for beta testing; they offer several ways for testers to submit feedback.
Product management and marketing should be working together to identify lists of people who would be interested in testing. The team should also work together to create a plan for receiving and triaging feedback into high/low priority issues that can be scheduled for development, either before or after the next public release.
What to Test For
Having an experienced exploratory tester on the team pays dividends because they understand all the different ways a user could encounter problems. This basic list from Phil isn’t exhaustive, but it should point you in the right direction. Situtions to test include:
- Airplane mode/no network
- Low battery mode
- Backgrounding and foregrounding the app
- Back button on Android
- Force quitting app and restarting
- App is never quit (memory issues?)
- Screen rotation
- Large text size in Settings
- Installing update over old version
- Saying No to permissions when installing app
- Dark Mode on iOS
- New phone/install for exstisting account pulls data from your server
- Accessibility features / screen readers
If your application will integrate with another IoT device, thoroughly testing this connection is very important. Situtions to test include:
- Out of range
- Multiple connections (i.e., multi users)
- Pairing, including with multiple devices in range
- Connecting then quitting and restarting
- Interference from other devices once connected
For more suggestions on what to test for, read Phil’s posts here on this blog. He has a long-running series on “More Ways to Test.”
Moving Forward
Finally, as table stakes, look to add a monitoring/crash reporting library like Crashlytics to the app. You don’t want to be flying blind after release, relying on your users to report quality problems.
Working with Phil has taught me there’s really no substitute for experience when it comes to high-quality exploratory testing, but his perspective opened my eyes to many potential issues I hadn’t considered. I hope this post will also be helpful to you.