Prompt Caching & Context Caching

The problem: paying to re-read the same text

Every time you call an LLM API, the model has to process the entire input from scratch — the system prompt, any tool definitions, prior conversation turns, retrieved documents, all of it — before it can generate a single new token. In an agent loop, most of that input doesn't change between calls: the system prompt is identical every time, the tool schemas are identical, and often large chunks of conversation history are identical to the previous call plus one new turn.

Without caching, you pay full price — in both tokens billed and compute latency — to re-process that unchanged prefix on every single call. For a long-running agent making dozens of calls per task, this redundant reprocessing can dominate total cost.

How prompt caching works

Prompt caching (also called context caching, depending on the provider) lets you mark a prefix of your prompt as cacheable. The first time that exact prefix is sent, the provider processes it normally and stores the resulting internal state (the attention key/value cache for that prefix, computed during the transformer's forward pass). On subsequent calls that start with the identical prefix, the provider reuses the cached state instead of recomputing it — skipping the expensive part of processing and jumping straight to generating from where the new content begins.

The practical effect: cached tokens are billed at a steep discount (commonly 90% cheaper than uncached input tokens) and processed with much lower latency, because the model isn't redoing work it already did.

Caching only helps on the prefix. If any token near the start of the prompt changes, everything after that point in the prompt can no longer match the cache — the structure is strictly a shared-prefix match, not a general diff.

Where this matters most for agents

Prompt caching has an outsized impact on exactly the kind of workloads agents produce:

Long tool-use loops. A ReAct agent re-sends its entire history — system prompt, tool schemas, every prior Thought/Action/Observation — on every single step. Without caching, a 20-step agent run reprocesses an ever-growing prefix 20 times. With caching, each step only pays for the new content appended since the last call.

Large static context. Agents that inject a large document, codebase, or knowledge base into every call (rather than retrieving fresh content each time) benefit enormously from caching that content once and reusing it across many queries against it.

Multi-turn conversations with a heavy system prompt. Agent system prompts with detailed tool documentation and instructions can be thousands of tokens — caching that fixed overhead across an entire conversation session removes most of its recurring cost.

Cache invalidation: the part that bites people

Caches have a limited lifetime (commonly minutes, varying by provider) and are keyed on exact content match. A few practical gotchas:

Any edit to the cached prefix invalidates it. Changing a single word in your system prompt, even unintentionally through templating logic, breaks the cache for every call using that prefix until a fresh cache entry is written.

Caches expire on idle time. If an agent pauses for longer than the cache's time-to-live between calls, the next call pays the full uncached price again, even if the content is identical.

Caches don't transfer across different prompt structures. If two code paths assemble a logically-equivalent prompt in a different token order, they won't share a cache, even though a human reading both would say they're "the same."

The practical implication is that cache hit rate is a property of your prompt-assembly code's consistency, not something you get automatically just by having a cacheable system available.

Cached vs. uncached calls in a long agent loop

What's next

Caching optimizes how much you pay for what the model reads. A related but distinct concern is controlling the shape of what the model writes back — the next lesson covers structured outputs and JSON mode, for when you need the model's response to reliably parse as valid, schema-conforming data.