Speculative Decoding

The bottleneck: one token at a time

Autoregressive LLM generation is inherently sequential: to produce token 50, the model needs tokens 1 through 49 already decided. Each new token requires a full forward pass through every layer of the network. For a large model, that forward pass is expensive, and you pay its full cost for every single token — even easy, highly predictable ones like finishing the word 'the' or closing a bracket.

This sequential dependency is the core latency bottleneck of LLM inference. You can't simply parallelize token generation the way you parallelize training, because each token's computation genuinely depends on the previous one having been chosen.

The idea: guess first, verify in bulk

Speculative decoding sidesteps the bottleneck without changing the sequential dependency of the final output. It uses two models: a small, fast draft model and the large, slow target model whose output you actually want.

The draft model generates several candidate tokens ahead — say 4 or 5 — very cheaply, one at a time. Then the large target model does a single forward pass over the entire draft sequence at once (this is the key trick: verifying a sequence of tokens in one pass is much cheaper per-token than generating them one at a time, because the large model's forward pass can process multiple positions in parallel). The target model checks, token by token, whether it would have generated the same token the draft model guessed. It accepts every token that matches, and as soon as it hits a mismatch, it discards the rest of the draft and generates the correct next token itself.

Why verification is cheaper than generation

This only works because of how transformer inference computes probabilities. Generating N tokens one at a time requires N sequential forward passes, each dependent on the last. But scoring a fixed sequence of N tokens — i.e., computing what the model's probability distribution would have been at each of those N positions — can be done in a single forward pass, because the model can compute all N positions' representations in parallel once the token sequence is fixed. That single parallel pass costs roughly the same as one sequential forward pass, not N of them.

So the economics work out: cheap draft tokens plus one expensive-but-parallel verification pass beats paying the expensive cost N separate times — as long as the draft model guesses correctly often enough to make the bulk verification worthwhile.

Standard decoding vs speculative decoding

When the speedup is biggest — and when it isn't

Speculative decoding helps most on text that is easy to predict: common phrasing, boilerplate code, repeated structure, closing syntax. A small draft model can match a large model's choices often on this kind of text, so long runs of tokens get accepted per verification pass.

It helps less on highly creative, surprising, or technical text where the large model's next-token choice is genuinely hard to predict — a weak draft model will guess wrong frequently, forcing the target model to fall back to generating tokens one at a time anyway, and the overhead of running the draft model becomes pure waste. Choosing or training a draft model that's well-aligned with the target model's typical outputs (sometimes literally a smaller model from the same family) matters a lot for real-world speedups.

What's next

Speculative decoding speeds up inference without touching what the model is allowed to say. Constitutional AI and alignment techniques go in the opposite direction — shaping what a model should and shouldn't say in the first place, through explicit principles and self-critique rather than just human-labeled examples. That's the next lesson.