There’s nothing like having real users using your system to unearth issues and ways of working with the software system that you hadn’t considered. Getting your app into their hands as soon as possible is highly recommended. Here are a few recent stories that illustrate the utility of user testing.
A Different Way of Sharing
Users found that the system was losing track of some of the work they had done, and we were having problems reproducing the problem. After asking for more detail, a few words stood out: “signing in and out of accounts.”
The system was still in the very early stages, and a planned enhancement was to allow easy sharing of work. Until this feature was done, the users had found a workaround and were swapping login details so they could work on the same piece this way. All our attempts to reproduce the issue until then had been logging in/out as the same user. But, once we added doing this to different accounts, the problem was soon reproduced and the fault tracked down: not all the state was fully cleared.
A “Sharing Accounts” scenario was added to my ever-growing list of test ideas.
The Early Bird
A user had an issue where a list was showing empty but would then show the correct values on a refresh. The issue was only happening for this particular user. After getting more details about this user, it turned out he was the first user of the system every day. With this information, we were able to track the problem down to a “Cold Start” situation with the server that we hadn’t noticed since we were all busy making deployments and testing them out all day.
A “First User of the System at the Start of a Day” was added to my test scenario list.
The Not-Noticed Bug
Users not reporting an issue can also be a guide as to how they are using the system. If you find a bug that the users haven’t reported, it could mean two things:
- They noticed it but didn’t report it — it could be they didn’t know they were meant to or it wasn’t really a problem for them.
- They weren’t using that part of the system, in which case they would never come across the issue. Asking questions or checking analytics can indicate if that part of the system is being used or not. If not, then you can ask why that part of the system is not useful to them.
The Secret Project
User testing to give you feedback is incredibly powerful. That’s a lesson I learned the hard way early in my career. The company I was working for decided to build a new product but was worried details would leak out and our competitors would make their own system. So we built it in secret. It looked great and was solid — and the users hated it. We hardly sold any copies.
What lessons have you learned from users getting their hands on your programs? Let me know in the comments.