A Token Cost Spiral We Didn't Catch Until the Invoice Arrived

The invoice arrived and it was roughly four times the previous month's spend. Nobody had flagged anything unusual before then, because the dashboard everyone was watching tracked total requests, not total tokens, and total requests had barely moved. The spike was hiding entirely inside token volume per request.

What was actually happening

A retry loop, added weeks earlier to handle occasional output schema failures, had no cap on retry count and no exponential backoff. For a specific input pattern that was increasingly common — a certain kind of malformed user input — the model would fail validation, retry with the full conversation history re-sent each time, fail again, and retry again, sometimes six or seven times per original request. Each retry resent the growing context, so the token cost per failing request wasn't just multiplied by retry count — it compounded.

A request-count dashboard told us everything was fine while the thing actually costing us money was happening entirely inside the requests we were already counting as normal.

How we actually found it

Not through monitoring — through the invoice, which is the expensive way to find out. Once we went looking, it took about twenty minutes to find in the logs: a cluster of requests with retry counts in the double digits, all tied to the same malformed-input pattern, all re-sending a growing context on every attempt.

• Added a hard retry cap (three attempts) with exponential backoff, and a circuit breaker that fails fast and logs instead of retrying indefinitely.
• Fixed the retry to resend only the minimal necessary context, not the full accumulated conversation history.
• Built a token-cost-per-request dashboard as a first-class metric, not a derived afterthought from the billing page.
• Set a daily spend alert threshold that would have caught this spike on day two instead of week six.

The lesson that stuck with me wasn't about retries specifically — it's that any dashboard tracking request count instead of token count on an LLM feature is watching the wrong variable. Request count tells you about traffic. Token count tells you about cost, and cost is the thing that actually spirals silently.