Sliding Window Attention

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The quadratic cost of full attention

Standard self-attention computes an interaction score between every pair of tokens in a sequence, giving it $O(n^2)$ compute and memory cost in sequence length $n$. For short sequences this is fine; for long ones — document-length contexts, long conversations, codebases — it becomes the dominant bottleneck in both training and inference. Sliding window attention is one of the simplest and most effective fixes: instead of letting every token attend to every other token, each token is restricted to attending only to a fixed-size window of nearby tokens (e.g., the preceding $w$ tokens), reducing the cost to $O(n \cdot w)$ — linear in sequence length rather than quadratic, as long as the window size $w$ stays fixed.

How information still propagates beyond the window

A natural worry: if token 1000 can only attend to tokens 900-1000, how does information from token 1 ever reach it? The answer is that sliding window attention is typically stacked across many transformer layers, and each layer's window effectively extends the reachable range by composition — information from token 1 can reach token 900 by layer 1 (within its window), then reach token 1000 by layer 2 (since token 900's layer-1 representation, now informed by token 1, is within token 1000's window), and so on. Across $L$ layers with window size $w$, the effective receptive field grows to roughly $L \times w$ tokens — similar in spirit to how stacking convolutional layers with small kernels expands the effective receptive field in a CNN.

sliding_window_mask.py

python

import torch

def sliding_window_mask(seq_len, window_size):
    """1 = allowed to attend, 0 = masked out. Causal + local window."""
    mask = torch.zeros(seq_len, seq_len, dtype=torch.bool)
    for i in range(seq_len):
        start = max(0, i - window_size + 1)
        mask[i, start:i + 1] = True   # can only see itself and the previous (window_size-1) tokens
    return mask

mask = sliding_window_mask(seq_len=10, window_size=4)
# Row i has at most 4 True entries, all at positions <= i (causal + local)
# Attention scores outside the window are set to -inf before softmax:
# attn_scores = attn_scores.masked_fill(~mask, float("-inf"))

Attention span strategies compared

Strategy	Compute scaling	Effective receptive field	Used in
Full (dense) attention	O(n²)	Entire sequence, every layer	Standard transformer, short/medium contexts
Sliding window attention	O(n·w)	Grows with depth: ~L × w across L layers	Mistral, Longformer (local component)
Sliding window + occasional global tokens	O(n·w) plus small global term	Local plus a few long-range anchor points	Longformer, BigBird
Attention sinks + sliding window (StreamingLLM)	O(n·w), bounded cache	Local window plus permanent early-token anchors	Streaming/long-running generation

Sliding window attention composes well with the KV cache

Because each token only ever needs to attend to a bounded window of recent tokens, sliding window attention also bounds the KV cache size needed for efficient autoregressive generation — older key/value pairs outside the current window can be dropped from the cache entirely (combined with care around attention sinks, as in StreamingLLM), which is a major practical reason it's attractive for deployed long-context or streaming inference, not just for training efficiency.

It's a genuine tradeoff, not a free efficiency win

Restricting attention to a local window does lose information compared to full attention — any task requiring a token to directly relate to something far outside its effective receptive field is harder for a sliding-window model, even accounting for the cross-layer composition effect. Model designers that adopt sliding window attention (such as Mistral) typically report it as a favorable tradeoff for most practical tasks given the compute savings, but it is a real capability tradeoff, not a strict improvement.

What's next?

Sliding window attention is one specific, regular pattern within a much broader design space of sparse attention patterns — some models use global anchor tokens, strided or dilated patterns, or learned/data-dependent sparsity instead of a fixed local window, each trading off compute savings against a different shape of information loss.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →