Results From Using AI-Assisted Testing on a Real Codebase

A client's tracking library had maybe 40% test coverage and nobody owned writing more. So I pointed an AI test-generation tool at the repo, gave it the existing test files as style reference, and let it churn out coverage for three weeks before I actually looked at what it produced.

The honest number: coverage went from 40% to 78%. That sounds great until you look at what kind of tests got added. About two-thirds of them were shape assertions — this function returns an object with these keys — which catch almost nothing. The other third were genuinely useful edge-case tests, several of which caught real bugs in how we handled malformed UTM parameters.

What it actually caught

• A null-pointer-style crash when a GA4 event payload was missing the client_id field entirely.
• An off-by-one in a pagination helper that only showed up with exactly 25 items, the page size.
• A timezone bug where a date-bucketing function used local time instead of UTC, visible only in tests run outside Pacific time.

AI-generated tests are great at finding bugs you'd have found eventually, and nearly useless at finding the ones that actually hurt.

What it didn't catch

None of the generated tests caught the actual production incident that started this whole exercise — a race condition in a webhook handler under concurrent load. That requires understanding the system's concurrency model, not just reading function signatures and generating plausible inputs. The tool has no model of time, ordering, or external state, so anything that depends on those stays invisible to it.

The real value turned out to be velocity on the boring 60%, not quality on the dangerous 10%. If you're using AI test generation expecting it to replace thinking about failure modes, you'll be disappointed. If you're using it to stop spending afternoons writing shape assertions, it's a genuine win — just budget a human pass for anything involving timing, concurrency, or external state.