AI-Generated API Documentation, Reviewed Honestly Against the Real Thing

We had a hand-written reference doc for an internal events API, written by the engineer who built it, polished over two years of support tickets fixing whatever was unclear. I fed the same codebase to an LLM and asked for documentation, then compared the two outputs side by side.

The generated version was faster to produce — minutes instead of the original multi-week effort — and structurally it was fine. Every endpoint, every parameter, every response shape was accurately described. If you'd shown me both docs without context, the AI version would have looked like a perfectly competent first draft.

Where it fell short

The gap was entirely in the parts that came from scar tissue, not from the code. The hand-written doc had a callout box explaining that the webhook retry behavior looks idempotent but isn't under a specific race condition, with a one-line workaround — knowledge that existed only because someone got paged about it once. The generated doc had no equivalent, because nothing in the code itself hints at that failure mode; it only shows up under production concurrency.

AI documentation describes what the code does. Good documentation describes what the code does wrong, and nothing in the code tells you that.

• Accurate on parameters, types, and response shapes — essentially flawless here.
• Missing every "gotcha" callout that came from an actual production incident.
• No sense of which endpoints are deprecated-but-still-used versus actively recommended.
• Wrote confidently about rate limits that were actually wrong, inferred from code comments rather than the real enforced values.

My actual workflow now: generate the first draft with AI, because it saves the tedious parameter-by-parameter transcription, then have whoever owns the API add the scar-tissue callouts by hand. That combination beats either approach alone, and it's roughly a third of the time the fully manual version used to take.