Article summary
Agent loops are in fashion these days, so I thought I’d talk about one of my favorite ones. Strangely, designing a useful agent loop turns out to be eerily similar to designing feedback loops in “pre-AI” software engineering. Or any sort of engineering, come to think of it. But let’s not overthink this!
As a software engineer, and especially as a consultant, I often have to deal with the competing demands of craftsmanship and the pressure of getting features out on time. In this respect, agentic coding has been a powerful tool and a great accelerator. All the tools for running tests, auditing security and code quality already exist, and they can live in our CI pipelines. It’s easy to turn that pipeline into a loop by connecting the two ends with an agent driving it.
What I like about this loop is that it’s surprisingly cheap, flexible, easy to set up, and unreasonably effective. It also works in almost any kind of project with a CI pipeline. The simple idea is to take a standard CI pipeline and make it much longer and much stricter than you ever would on a typical software project. And you drive all fixes with an agent. So simple.
3 Principles
There are three principles of this pipeline construction:
- Every step in the pipeline should be orthogonal to all the others. This way, each step gives you a lot more information about the correctness of your application.
- Prefer self-contained, deterministic, and domain-specific software for each check. In a long pipeline, a series of non-deterministic steps increases the odds of the overall pipeline breaking for reasons unrelated to application quality.
- Keep it linear, with earlier checks faster than later checks. This makes your agent focus on a single problem at a time.
They are principles and not rules because they are something to aim for. There are lots of checks, like running your tests in a random order, that can make sense and go against a principle. And I bend these principles all the time.
An Example
The most common application I encounter is a web app with an API and infrastructure assets (like Dockerfiles, Terraform, and Kubernetes YAML). We can work with an example of that:
[compilation/type check] -> [secrets check] -> [linting/formatting check] -> [unit tests] -> [integration tests] -> [browser or E2E tests] -> [schema migration check] -> [dependency security check] ->
-> [static code analysis] -> [container build] -> [infra security check] -> [LLM code review]This is very linear and very long. You would not want to make this pipeline pass on each feature without help. This would be set up to run on a branch. On GitHub, we run this on pull requests using GitHub Actions.
In my simple setup, the agent controls the pipeline locally via the GitHub CLI. You can definitely make a skill for this, but you can also typically just say, "Iterate on this PR until green; read all PR comments and address them if valid," and move on to doing something else until it's done. The most advanced Anthropic and OpenAI models know exactly what to do.
Various Checks
The [compilation/type check] is typically just an install and build. For example, in a web app, this could just be a "pnpm install" and a "pnpm build".
For the [secrets check], I like to use Gitleaks or Trivy. This helps detect any sensitive data that you may have checked into your project, and honestly belongs as a check that runs in a Git pre-commit hook. But hooks don't always run, and at least you can catch it here.
The [linting/formatting check] is typically an ESLint/Prettier combo. I've really enjoyed Biome recently as it's quite a bit faster, which can matter in a pipeline this long.
The [unit tests], [integration tests], and [browser or E2E tests] are run in the order of complexity. We want to stay true to the Test Pyramid model, where we have more fast unit tests and far fewer slow E2E tests. We try to leverage unit tests to catch more of the bugs that we can conceive of, and the slower tests should cast a wide net to catch the ones we didn't expect.
The [schema migration check] should attempt to run the schema migrations against a real database. This might already happen during integration testing against an empty database. But in this step, if possible, try to run a "dry-run" migration against some real data (avoid actually applying it to the environment, though). There's a huge difference between applying a migration to an empty database and running it against one with real data.
Always, but especially in this time of non-stop npm package hijacking, it's very important to scan your web app dependencies. In the [dependency security check], we do a vulnerability check over all the package lockfiles and fail if any vulnerability exists. Note that sometimes making this step succeed involves major application surgery, so don't include it if you're not comfortable with your coding agent doing that. I like OSV-Scanner for this step.
Quality Check
It gets interesting for the [quality check] as this is a little subjective. What fits in here is something like SonarQube (for comprehensive checks) and/or react-doctor (if you are writing React). I've been having a lot of luck keeping it simple and using the "cognitive complexity" metric, which is available in Biome and also in ESLint via the sonarjs plugin. If you have documentation, there are also tools that can force the agent to write better documentation like the Documentation Readability Analyzer (which especially helps Opus/Fable write better technical documentation).
For situations where you are building containers, you'll want to check that you can build them in your pipeline in a [container build check]. This is a simple check to see if you can build a container, if you're building one. If you're packaging some other way, this step should test that instead.
The [infra security check] step is a place to check all of the deployment artifacts. Trivy can scan your container as well as your Terraform and Kubernetes files. Again, fixing some infrastructure misconfiguration can increase the blast radius of a pull request dramatically, so review these changes carefully.
For the last [LLM code review] step, I like to arm an LLM with the GitHub CLI and allow it to comment directly on the pull request. This already breaks one of my principles, but I bend my own rules here because it's paid off so much. It's important that the model doing the review is different from the model writing the code and that it's reasonably priced. Cursor's model is very good for this, but the landscape is constantly changing. You can see the complete prompt and isolated workflow in this example repo.
Setting It Up
You can easily trust a coding agent to set this up for you. You can even just copy and paste this blog post for it to look at. In the course of setting up this pipeline, your agent will have plenty of feedback, so it will have no issue iterating until the pipeline is correct.
Since the agent can deliver a lot more code than you can reason about, this pipeline adds some gates that make it easier to focus on higher-level problems (like, should this feature exist? should it work like this? can people use it?). It also helps with the pace of reviewing the pull requests of others on your team.
The initial run of an agent loop like this on brownfield code can take a while and can introduce instability, so make sure to watch it closely. It's entirely possible you'll want to hold off on some changes or do extra testing. Once the pipeline is green, keeping it green is a lot less work. Just like the CI pipelines of old.