Surviving an Infrastructure Upgrade

In any moderately complex application, there comes a time for major upgrades. These upgrades are often deprioritized until some business case knocks on your door to make the update more urgent, which can raise insanity levels and make jumping this hurdle seem insurmountable. After having gone through this process several times on various projects, I wanted to share some of my strategies to deal with such a potentially vicious beast.

1. Establish a Solid Baseline

Before you map out your journey to somewhere new, you need to define your starting point. With software, that starting point is usually a commit hash. If you are doing a major framework update, it may take days or even weeks to accomplish the upgrade. Limiting the number of variables during this transition will help you maintain some level of sanity.

2. Develop a Validation Strategy

At the outset of a major framework update, it’s crucial to consider how you will validate that the resultant system works. This may seem like a given, but when you are uprooting a complex system, the measures you have in place to maintain it throughout small changes may not be sufficient to catch any large-scale system failures caused by major rework.

Furthermore, with a large upgrade, you may not be able to reassemble the system to run system-level tests for quite some time. During this large-scale outage, you may need to rely on lower-level testing to validate the individual pieces.

This is where automated tests at the unit and mid-range integration level become crucial. If you don’t have these levels of validation in place, you may end up feeling much more pain and much less confidence in reaching your goal.

3. Keep a Journal

Doing a framework or infrastructure upgrade may involve monkeying a bit with dependencies and the way your system is pieced together. Documenting these higher-level changes is therefore crucial to keep track of where you are and where you have come from.

These records of change can give you and your team insights about what went into the effort. They can also instill confidence in your progress even when you end up in difficult situations along the way.

4. Keep an Eye on Scope

Any good plan requires a definition of the overall goal, including details on the scope of changes required. It is likely that you will have to reassess the scope as you go. Updating one dependency may cause several others to be updated as well, and you may discover some great new tools that you can benefit from when you upgrade.

During the upgrade, you may decide to increase or decrease the scope of your goal. Increasing scope may require more calendar time to complete the upgrades at hand. Defining this scope will help you make tough decisions about what else to bite off (or not) along the way.

5. Deliver Value

In the end, you need to deliver value to your team and your business. Make sure you identify the value being added from the get-go so you can get buy-in from all those involved and affected by the changes you are making. If you can’t sell the value, then it may not be worth the investment.

Keep track of any additional value that you discover as a side-effect or add deliberately along the way. It will be very important to you, your team, and any stakeholders to remember how rewarding the upgrade will be as you slowly and methodically march toward completion.