Attention Sink & Streaming LLMs

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The problem with streaming generation past the trained context length

Deploying an LLM for long-running, streaming interactions — a chatbot session that runs for hours, or continuous log analysis — runs into a hard constraint: the model was trained on sequences up to some maximum context length, and naively letting the KV cache grow without bound, or crudely evicting the oldest tokens once a length limit is hit, both cause serious problems. Unbounded growth eventually exhausts memory; naive sliding-window eviction of the oldest tokens, it turns out, causes a surprising and severe collapse in generation quality, far worse than the mere loss of distant context would predict.

The surprising discovery: a few initial tokens matter disproportionately

Xiao et al.'s StreamingLLM work (2023) found the cause: a small number of tokens near the very beginning of a sequence — frequently including just the first one or few tokens — receive an unusually large share of attention weight from nearly every later token, regardless of their actual semantic content. These tokens were named attention sinks. The hypothesis for why this happens: the softmax function in attention always needs to distribute weight that sums to 1 across all tokens, but when a query token has no sequence position it needs to attend to strongly, the model has learned to dump the resulting 'leftover' attention mass onto a small set of fixed, always-present early tokens rather than letting it spread arbitrarily — a kind of attention 'release valve.' Evicting these specific tokens during naive sliding-window truncation breaks this mechanism and causes attention score distributions, and generation quality with them, to degrade sharply.

streaming_cache_layout.txt

text

Naive sliding window (evicts oldest tokens as window fills):
  [ token1 token2 ... tokenN-W ] [ tokenN-W+1 ... tokenN ]
    ^-- evicted once window fills      ^-- kept, sliding window
  Problem: evicting token1 (an attention sink) causes quality collapse.

StreamingLLM cache layout (keeps sink tokens permanently):
  [ sink tokens: first few, NEVER evicted ] [ sliding recent window ]
    e.g. tokens 1-4                           e.g. last 1020 tokens
  Cache size stays bounded; attention sinks stay available at every step.

StreamingLLM: keep the sinks, slide the rest

The fix is simple once the cause is identified: keep a small, fixed number of initial tokens' key/value pairs permanently in the cache (regardless of how old they get), and apply a normal sliding window eviction policy to everything else. This bounds memory use (the cache size stays fixed rather than growing with sequence length) while preserving the attention sink mechanism the model relies on, allowing stable generation quality over sequences far longer than the model's original training context length — in the original paper, stable generation was demonstrated over millions of tokens using models trained with much shorter context windows.

Approaches to long/streaming generation

Approach	Cache size	Quality over very long sequences	Requires retraining?
Unbounded cache	Grows without limit	Good until memory exhausted	No
Naive sliding window	Fixed, bounded	Collapses sharply once sink tokens evicted	No
StreamingLLM (sink + sliding window)	Fixed, bounded	Stable over very long streams	No — works on existing trained models
Fine-tuned/extended context length	Depends on new max length	Good up to new trained length	Yes — requires additional training

An inference-time fix for a training-time quirk

One of the more notable things about StreamingLLM is that it requires no retraining or fine-tuning at all — it's purely a cache-management policy applied at inference time, exploiting a behavior (attention sinks) that emerged naturally during ordinary pretraining. This makes it unusually cheap to adopt compared to context-extension approaches that require additional training runs.

It extends generation length, not effective memory

StreamingLLM keeps generation stable over very long streams, but it doesn't give the model genuine access to information from the evicted middle portion of the conversation — tokens outside the sink and the sliding window are simply gone from attention, not summarized or remembered in any form. For applications that need the model to actually recall something said far earlier in a long session, this technique alone isn't sufficient; it solves stability, not long-range recall.

What's next?

StreamingLLM's sliding window is one specific, fixed pattern of restricted attention; a broader family of techniques generalizes this idea deliberately at training time rather than only as an inference-time patch, restricting which tokens each position attends to in order to make long-context processing cheaper from the start.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →