Article summary
It’s always risky starting a large software rewrite. There’s significant risk in replacing a system that has been in use for a long time, as plenty of hidden institutional knowledge may have been lost.
In 2019, I began work on one of these rewrites. This InsurTech project was for a regional insurance carrier and was a portal for both independent agents and policyholders. Parts of their system were built on some pretty old technologies, which was limiting their growth. They wanted to offer an improved experience for their clients, but that wasn’t possible with the existing setup.
Risk Areas
One of the major limiting factors of the system was the web portal technology. The webpages were rendered via CGI processes written in RPG to generate HTML and served on an AS/400. The data display was tightly coupled to the data fetching, which constrained the logic to a particular set of U.S. states where they offered insurance.
In order to expand their business into new states, they would need to separate those two things. The plan was to create some new API endpoints on the AS400 to support the new web portal we were building.
Naturally, as we were working with a client in the insurance industry, we wanted to take a page out of their book and focus on reducing risk. One of the riskiest areas was implementing the new API endpoints on the AS400. Not many people were able to make changes to that system, so that put the success of the project at the hands of some (already) extremely busy engineers. In addition, when implementing new functionality, there’s a high probability of introducing new bugs. The existing web portal was already battle tested over the years, but there was no guarantee that the new APIs would behave the same way.
Reducing Risk
Our solution was to use their frontend as our backend! We were able to utilize web scraping technologies to fetch data from their web portal, in the same HTML format that had been served for years. This reduced the risk in two major ways:
- All team members were able to build out new features and fetch data without relying on the few engineers who could add API endpoints to the AS400 server.
- We didn’t need to reimplement the tricky business logic on the AS400 and possibly introduce new bugs.
We built a highly extensible React frontend that communicated with a Node.js server we maintained. The server would proxy requests to their AS400 server to fetch appropriate data and parse it into easy-to-use domain models that could be used throughout the application.
Additionally, we built a validation layer to ensure that the data parsed from the HTML was in an expected format. Finally, we architected the server to allow plugging in different adapters to fetch data below the data validation layer. This means that when the AS400 is eventually replaced, it will be easy to introduce a new adapter to fetch data from the appropriate provider.
Software rewrites are risky, but there are always some ways to reduce that risk. When looking at a rewrite, start to evaluate the risks, and think outside the box to come up with some solutions.