On a recent project, I watched a team spend the better part of a week tracking down a single regression. The pipeline was a long-running simulation suite. Each test run took hours. There were dozens of commits to check, and the only way to check them was manually: run the sim, wait, evaluate the result, pick the next commit, repeat. By the time they found the offending
change, the team had burned days on a problem that felt like it should have been automatable. It was. But the automation would have required thinking through two problems separately, and most teams conflate them into one.
The first problem is finding where a regression was introduced. The second is understanding why it happened. Both feel like the same problem when you’re in the middle of a painful manual bisect. They’re not, and treating them as one produces tools that handle each poorly. Separating them is where the real design work starts.
Finding vs. Understanding
Finding a regression is a binary search problem. You have a range of commits, a pass/fail signal, and a goal: narrow the window to the single commit that introduced the failure. Each round probes an evenly spaced commit and cuts the remaining range in half. O(log N) rounds, regardless of domain.
The only piece that varies by domain is the verdict function. Does this commit pass or fail? That’s the part you plug in through a strategy interface. Whether you’re running a hardware simulation, a model training pipeline, or a financial backtest, the search algorithm is the same. It’s a commodity, and it’s fully automatable.
Understanding is a different kind of problem. Once you’ve located the offending commit, you need to know what changed and why it caused the failure. That requires context, domain knowledge, and judgment. It’s not a search problem, it’s an analysis problem, and that’s where AI earns its place.
The team I was watching didn’t have either tool. They were doing the binary search by hand and relying on intuition for the analysis. Automating the search is the easier win, but getting the analysis layer right is where the real leverage is.
Adding AI Directly to the CLI
Once you’ve automated the search, the natural next step is to add AI analysis to the same tool. The most common approach is to wire an LLM directly into the CLI: embed a system prompt, call the model API, and surface the response inline. It ships quickly and feels like a complete solution.
But, with this approach, the problems tend to surface gradually. The tool now has an opinion about which model to use, which means swapping models requires a new release. Prompt changes (and there will be many as you learn what actually produces useful analysis) also require a new release. Every engineer who runs the tool is independently calling the same model with the same data, with no opportunity to deduplicate those requests. And there’s no natural place to add caching, rate limiting, or access control. Each issue is manageable on its own. Together, they make the AI integration hard to evolve and expensive to maintain.
Treating AI Analysis as a Stable API
A more durable approach is to give the AI analysis layer its own versioned contract and expose it as a service. The CLI calls an endpoint. It has no knowledge of which model powers it, how it’s prompted, or how responses are handled on the other side.
POST /analysis/compare
{
"baseline_id": "<id>",
"candidate_ids": ["<id>", "<id>"],
"focus": "optional free-text to steer the analysis"
}
Response:
{
"summary": "<AI-generated comparison>",
"metrics": [...]
}
The focus field is the key piece. It lets an engineer steer the analysis at call time without ever touching a system prompt or releasing a new version of the tool.
What this unlocks is worth spelling out. You can swap the underlying model without touching client code. Identical requests can be cached. Model versions can be A/B tested transparently. Auth and rate limiting live in one place. And the team responsible for analysis quality owns the prompt independently from the team maintaining the CLI. Those are two different concerns, and they
can move at their own pace when they’re properly separated.
Two Binaries, Two Audiences
There’s one more separation worth making explicit. The headless runner that performs the binary search and the interactive tool that surfaces the AI analysis should be different binaries with different design contracts.
The headless tool has no prompts, no menus, and exits with a structured result. It’s built for CI. It runs unattended and hands off a clean artifact when it finishes.
The interactive tool is session-persistent, so you can close it and resume where you left off. It’s menu-driven and surfaces the analysis inline so an engineer can explore the failure without leaving the tool. It’s built for a human sitting down to investigate.
Trying to serve both audiences from one binary leads to flag soup and a confusing experience for both. A CI job doesn’t need a menu. An engineer investigating a failure doesn’t need the tool to behave like a headless runner. Keeping them separate keeps each one simple.
Where This Pattern Applies
This pattern applies beyond any one domain, but the same architecture fits any situation with expensive test runs measured in hours rather than seconds, a meaningful pass/fail signal per run, and a human who needs to understand the failure once it’s been located. ML pipeline regression hunting, game build validation, financial model backtesting, and drug discovery screening
pipelines all fit this shape.
The binary search is the same in every case. The analysis layer is where the domain expertise lives, and where the tool either earns its keep or falls short.
Getting the Design Right
Watching that team work through their regression manually, the bottleneck wasn’t effort. It was tooling. The search was manual when it didn’t need to be, and the analysis was ad hoc when it could have been structured.
The insight isn’t that you should add AI to your tooling. It’s that you should be deliberate about what problem you’re asking it to solve. In regression hunting, AI belongs in the analysis layer, not the search. Giving that layer a stable API contract keeps the rest of the system from coupling to it. Exposing a focus parameter gives engineers control without requiring infrastructure access.
The search algorithm is a solved problem. Getting the analysis layer right is what makes the difference between a tool that actually gets used and one that gets abandoned the next time the model changes.