We Added an AI Code Reviewer to Our Process, Here's What It Actually Catches

We ran an AI reviewer against every PR on a client's repo for about three months before deciding whether to keep it. The pitch was obvious: catch the boring stuff before a human has to. The reality was narrower and, honestly, more useful than the pitch.

What it was genuinely good at

It never got tired. It flagged an unhandled promise rejection in a PR submitted at 11pm on a Friday that two human reviewers had already approved — not because the humans were careless, but because they were scanning for logic correctness, not exhaustively checking every async call site. The AI reviewer has no concept of being in a hurry to approve and move on to the next review.

The AI reviewer never gets tired of checking the boring thing for the hundredth time. That's exactly where it earns its keep, and exactly where humans burn out first.

What it was consistently bad at

• Anything requiring knowledge of why a decision was made three sprints ago — it has no institutional memory.
• Flagging a change that's technically correct but violates an unwritten team convention, like a naming pattern that exists only in people's heads.
• Distinguishing a deliberate performance tradeoff from an accidental one — it flagged a memoization we'd intentionally skipped, twice, with the same explanation each time.

We kept it, but we changed its job description. It's not a reviewer anymore in our process — it's a pre-review linter with opinions. Nothing it flags blocks a merge by itself, and nothing a human approves gets re-litigated by it. That split avoided both the false confidence of trusting it fully and the wasted effort of ignoring it entirely.

The actual time savings came from a boring place: it cut the average time-to-first-human-comment on a PR by about a third, because the obvious stuff was already flagged before a person opened the tab. That's a real win. It's just a smaller and more specific win than the marketing suggested.