We Stuffed the Context Window and Latency Tripled, a Postmortem

The feature started with a tight, fast prompt: a user question, a short system instruction, maybe 800 tokens total. Over six weeks, every bug report got fixed by adding more context — more examples, more retrieved documents, more conversation history 'just in case it helps.' By the time I got pulled in to investigate a latency complaint, the average prompt was north of 14,000 tokens and p95 latency had gone from 1.8 seconds to 6.1 seconds.

Nobody noticed because nobody was measuring the right thing

Each individual addition got reviewed for whether it fixed the bug it was meant to fix, and it usually did — adding an extra example genuinely did improve that one edge case. Nobody reviewed the cumulative token count, because no dashboard tracked it. Latency crept up gradually enough that it looked like normal variance until someone finally plotted it against a six-week window and saw the straight line up and to the right.

Every context addition was locally justified. The sum of locally justified decisions was a prompt nobody would have approved if they'd seen the whole thing at once.

What we found when we actually audited the prompt

• Four retrieved documents were being included per query when the eval set showed one or two were sufficient for 90% of cases.
• Conversation history was included in full rather than summarized, so long sessions ballooned the prompt regardless of relevance.
• Three few-shot examples had been added for edge cases that newer model versions no longer needed examples for at all.
• Nobody had re-tested whether the original 800-token prompt still performed fine on a newer model generation — it did, almost identically.

The fix was a budget, not a rewrite

We set a hard token budget per request — 3,000 tokens, enforced in code, not as a guideline — and made every context addition justify its place inside that budget against the eval set, the same discipline that should have been there from the start. Latency dropped back to 2.3 seconds and the eval score moved less than a point. The lesson wasn't 'context doesn't help.' It's that context has a cost that compounds silently unless something is actually watching the total, and 'it helped one case' is never permission to add it unconditionally.