About seven months ago, I wrote about using AI for code reviews. At the time, it felt pretty novel. I had a GitHub Action that gathered up our Cursor rules, fed them to Codex, and posted review comments on pull requests (PRs), and I was also trying out Sentry’s AI review beta. I said something like 60–70% of the comments were worth reading, and that you still needed a person to sort the good ones from the noise.
A lot has changed since, and it’s going to continue to change.
The bigger change is what I stopped doing.
I’m not really writing my own code by hand anymore.
Six months ago I was still typing most of it, with an agent helping around the edges. Now I describe what I want, an agent writes it, and I read the result closely and course-correct. Almost everything I actually type these days is instructions to the agent. I’m hardly ever in a file editing it myself anymore. That’s a genuinely different way to work than it was at the start of the year, and it happened faster than I would have guessed.
It also puts a lot of weight on the review step. The reality of letting an agent write the code is that I’m not sitting down and thinking through every edge case. I’m still reading the code but being less hands-on with the codebase as a whole means it’s harder to internalize it. I’m relying pretty heavily on the AI review to surface those missed scenarios as well as to self-correct on bad patterns. That raised the bar for what I need the review to be, and it’s a bar we’re still climbing toward.
We refined our homegrown review.
We’ve mostly dropped the external review tools. The homegrown one does the job, and we can change it when it’s not performing well enough. When it misses something, or nags about the wrong thing, I can open the prompt and fix it that afternoon. A vendor’s beta doesn’t give you that.
We’ve moved away from the custom script for loading the appropriate Cursor rules and instead rely on the agent to identify which rules it should read based on the files that have changed. This means a little less code to maintain, which is always a plus, but it’s also not guaranteed to be as reliable.
We also tweaked the output of the review step to have it post a single comment with a list of all the must fix/should fix findings and suggestions that it identifies, rather than commenting on individual lines. The script for commenting on specific lines was finicky and quickly became a maintenance burden.
If the comment includes any must-fix findings (checked with a simple regex), the review bot will request changes on the PR, which forces the author to either address the issues or override the review with a specific label and a justification comment. If there are no must-fixes, it will approve the PR. We require an additional human approver too.
Turning the AI review into a blocking gate has driven many iterations on the review process over the last couple months. If we’re giving a robot the power to block a developer from merging work that another human has approved, we’d better make sure the robot is only doing so when there are actually real issues.
The hard part is consistency.
The most frustrating problem early on was that the review wasn’t consistent from one run to the next. Run it twice on the same PR and you could get two different sets of findings. Something would come back as a hard must-fix on one run and go completely unmentioned on the next. When the review is a blocking gate, that’s a real problem. It’s hard to trust a reviewer that doesn’t say the same thing twice.
The worse version of this was when it kept finding brand-new must-fixes every time it ran. You’d address everything on the list, push the fix, and the next run would come back with a fresh set of blockers it had never raised before. The author does the work, re-runs, and there’s a whole new list waiting.
Most of that came from each run starting cold with nothing but the diff. So we started giving it more to work with. The PR description goes in now, so the review understands what the change is actually trying to do. Its own earlier reviews on the same PR go in too, so it can build on what it already said instead of re-deriving everything from scratch. We also feed it the review comments and the author’s replies, including the justification on any override. This ensures it re-raises any unaddressed previous findings, and it’s able to downgrade findings that the user has posted a good justification for.
There’s a slower loop that runs on the prompt itself. When a human reviewer catches something the bot let through, we don’t just fix the code. We write that specific miss into the prompt. Over months that’s turned into a fairly opinionated document that encodes the exact ways our team has actually shipped bugs. By now it reads like a checklist of the things we keep getting wrong, and it grows every week.
Run it locally first.
One thing I’ve folded into my own routine is running the review before I push. I set up a skill with Claude that pulls in the review prompt so it can run the same analysis that CI does. If it turns up a must-fix, I fix it and re-run right there on my machine. Without that, the loop is slower and more annoying: push, wait for the gate to fail, fix, push again. That churn adds up over a day, and catching most of the blockers before they leave my machine cuts it way down.
It’s still a work in progress.
I don’t want to leave the impression that we’re satisfied with our AI review process. There’s a persistent problem where re-running the review on a branch surfaces must-fix findings on code the latest change didn’t actually touch, and getting it consistent across runs is still something we’re chasing. The reality is most of our PRs end up having the AI review overridden right now.
The most recent challenge is the model itself. We switched the model behind the reviews from a hosted commercial one to a self-hosted open-weights model, and it’s been a step down so far. The output isn’t formatted as consistently, and the findings are more scattered. A prompt tuned against one model doesn’t just carry over to another. I rewrote a good part of ours to fit how this one actually behaves. That has helped, and the reviews are in a better place now, though still not perfect. Every time the model underneath changes, a lot of the tuning has to be redone, and that’s just part of the work now.
The caution I ended the original post with holds more than before: this works because someone maintains it, constantly. None of it is set-and-forget, and I don’t expect it ever will be.