April 8, 2026 · 5 min read
An LLM Caching Strategy That Cut Our Costs by 60%
A client's LLM bill was growing faster than their usage. The fix wasn't a cheaper model, it was caching the right things at the right layer.
The client's instinct when the LLM bill got uncomfortable was to downgrade to a cheaper model across the board. We ran the numbers on a sample of their actual traffic first, and it turned out roughly 70% of their requests were either exact or near-exact repeats of prompts they'd already processed — the cost problem wasn't the model, it was paying full price for the same question over and over.
The product was a document Q&A tool where many users asked near-identical questions about the same shared document set. A cheaper model would have reduced cost per call while degrading answer quality on every call, including the 70% that didn't need to be recomputed at all.
What we cached and where
Request comes in
-> exact prompt + context hash match in cache?
yes -> return cached response (no LLM call)
-> no exact match, but prompt prefix overlaps a cached context?
yes -> use provider prompt caching for the shared prefix
-> no match at all -> full LLM call, cache the result- An exact-match cache keyed on the normalized prompt plus the relevant document context hash, for genuinely repeated questions
- Provider-level prompt caching for the large shared document context that appeared across many different user questions
- A short TTL on cached results for anything tied to data that could go stale, to avoid serving outdated answers indefinitely
The cheapest LLM call is the one you don't make. We were solving a caching problem with a model-pricing solution.
The result
Total LLM spend dropped about 60% within the first month, with no change to the model and no measurable change in answer quality, because the requests being served from cache were genuinely identical to ones already computed. The remaining 30% of truly novel requests still hit the full-price model, which was the right place for that spend to go. Chasing a cheaper model would have cut costs by maybe 40% while making every single answer worse — the caching approach cut more and made nothing worse.