Does anyone need this product? Are they willing to pay for it? If you have ever been part of a startup product team, then you have undoubtedly tried to answer these questions. In a former role as a product manager, I spent a lot of time assessing product-market fit. So, I took any chance to better understand the ability of my product to satisfy the needs of my intended market. Conducting a beta release provides that opportunity.
If you’ve not been through a beta release phase before, you might wonder how to prepare your development team for future work. Here, I’ll share how I categorized and prioritized my product backlog during a beta release. But first, let me review some characteristics of the specific release type I’m talking about.
Development Stages and Release Phases
There are stages of development and phases of release. The early stages of software development primarily focus on testing. These three stages are generally known as Alpha, Beta, and Gamma.
Releases can occur during phases of the testing and development period. In an alpha release, the intended users are internal team members. In contrast, beta releases target some segment of the targeted market.
Closed or Open Beta?
Beta releases are either closed or open. Whichever you choose depends on how many people you feel comfortable letting use your feature or product in its current stage.
In closed beta, the organization selects and invites a manageable number of users to exercise the system. The system is feature complete enough to test important workflows to gather feedback. However, it is usually not ready for heavy user traffic. It’s common to compensate users who accept the invitation in appreciation for their time in the system.
An open beta doesn’t restrict the number of users. This is because the system is stable enough to handle heavy usage. With more users in the system, presumably, the product team can expect feedback from more segments of their intended market.
Prepare to Iterate
In a beta release phase, teams are primarily interested in rapid experimentation. This means they can learn as much as possible about product-market fit.
Before releasing a beta version, the product team should try to anticipate the types of feedback coming their way and how it will allocate resources accordingly. This pre-planning will help the team quickly capture and break down work. With the work defined, the team can iterate on what they delivered in the beta release.
In my case, our product team used a spreadsheet to brainstorm categories that we expected feedback to fall into. We refined the categories and used the list to structure our development backlog.
Categorize Items
The backlog will help the team navigate the beta release by having work items at the ready. Our backlog consisted of items that fell into the following categories:
- Critical bugs. In my project, we defined these as defects that obstructed the testing process.
- User-facing features. The features that beta users interacted with.
- Infrastructure that supports the user-facing features of the system. This work would allow the product team to experiment quickly without taking down parts of the system. Work in this category was tied to being able to measure usage and adoption and isolate problems.
- Cross organizational functions. Things that supported the horizontal layer of the business such as legal, privacy, and security concerns. System reliability fell into this category as well.
- Technical debt. Items in this category were largely the result of some change in scope, performance, scalability, and/or technology.
I created a label in Jira for each of the five categories. Then I tagged every work item with one of the labels. Every two weeks the backlog was sorted by label and exported to a file. This is the list that the product team used when we met with internal stakeholders to prioritize the next sprint or two of development work.
Prioritize Items
Ours was a closed beta. We had just over 100 active daily users. With so few people in the system, we didn’t have much actual user data to help us prioritize our backlog. So, we decided to use a simple backlog prioritization framework called ICE to help us figure out which work to go after and in what order.
Startup Advisor, Sean Ellis, popularized this scoring model. It stands for Impact, Confidence, and Ease. In a nutshell, you assign a value to each metric for the item being discussed. The three values are multiplied together to get the priority score. The items with the highest score will presumably deliver the most value if implemented.
We took each work item in the backlog and assigned a number to it. We used a scale of 1 to 5, one being low and 5 being high.
- Impact – If implemented, we will better understand product-market fit.
- Confidence – Level of certainty in the Impact number.
- Ease – The amount of work it will take to implement.
What I liked about this framework is that we could quickly assign relative numbers to work items. Then in subsequent discussions with stakeholders and team members, we could quantify our backlog in terms of importance.
Conclusion
A beta release is one of the best opportunities to hear what the market wants and to shape what will eventually be delivered. Before releasing, think about the shape of work that could come in during this phase. And be prepared with some framework and method for prioritizing development work.