Chain-of-Thought Reasoning

The trick that shouldn't work but does

A language model generates one token at a time, each token conditioned on everything written so far. If you ask a model a hard math or logic question and demand an immediate one-word answer, it has to produce the correct result in a single forward pass — no scratch space, no do-overs.

Chain-of-thought (CoT) prompting changes the task: instead of asking for the answer directly, you ask the model to work through the problem step by step and give the answer last. Something as simple as appending "Let's think step by step" to a prompt measurably improves accuracy on arithmetic, logic, and multi-hop reasoning tasks — often dramatically, on problems the model gets wrong when asked to answer directly.

Why it works: tokens as compute budget

The core mechanism is simple: every emitted token is an opportunity for the model to condition its next prediction on more information. When a model writes out intermediate steps, each step becomes part of the context for the next one. This effectively gives the model extra 'thinking space' — a kind of external working memory spelled out in natural language — that a single terse answer doesn't have.

This is sometimes described as trading test-time compute for accuracy: generating more tokens before the final answer costs more inference time and money, but tends to produce better results on tasks that require composing several steps (arithmetic, multi-hop lookups, code tracing, planning).

From a prompting trick to a trained behavior

Early chain-of-thought was purely a prompting technique applied to models not specifically trained for it. Newer 'reasoning models' internalize this behavior: they are trained (often with reinforcement learning on verifiable tasks like math and code) to generate long internal reasoning traces before producing a final answer, without needing to be asked.

These models often allocate a variable, sometimes very large, 'thinking' budget automatically — spending more tokens on reasoning when a problem is hard and fewer when it's simple. The user-facing distinction matters: with a prompted CoT, you're nudging a general-purpose model's existing behavior; with a trained reasoning model, extended reasoning is the model's default mode, and the visible chain of thought is often a trained artifact of the underlying computation, not a free-form addition.

Prompted CoT vs. trained reasoning models

This is not the same thing as 'reasoning'

It's tempting to read a fluent chain-of-thought trace as a transparent window into how the model 'actually thinks.' That's an overstatement in both directions.

First, a model's final answer isn't guaranteed to follow from its stated steps — models can write a plausible-looking derivation and then produce an answer that doesn't actually follow from it, or reach the right answer via steps that don't causally explain how the underlying computation worked. The visible trace is text generated by the same next-token process as everything else; it's correlated with better answers, but it is not a verified audit log of an internal reasoning process.

Second, chain-of-thought doesn't give a model new capabilities it fundamentally lacks — it helps the model better use capabilities it already has by decomposing a hard single-step prediction into several easier ones. A model that has never learned to multiply large numbers won't reliably learn to by narrating its steps; it will narrate plausible-looking but wrong arithmetic just as confidently.

Where chain-of-thought helps most

CoT gives the largest gains on tasks that decompose naturally into sequential steps: arithmetic word problems, multi-hop question answering, symbolic reasoning, code tracing, and planning tasks with several dependent sub-goals. It gives little or no benefit on tasks that are essentially single-step lookups or pattern matches (simple fact recall, short classification), and can even hurt on tasks where a confident wrong first step snowballs — a phenomenon sometimes seen when a model locks into an incorrect framing early and then reasons its way to a confidently wrong conclusion.

Practical takeaway: use CoT (prompted or via a reasoning model) for genuinely multi-step problems, and skip it for simple ones where it only adds latency and cost without improving accuracy.

What's next

Chain-of-thought is one lever for getting more out of a model's existing weights at inference time. Mixture of Experts tackles a different problem — how to scale the number of parameters a model has without scaling the compute cost of every forward pass proportionally. The next lesson covers how sparse expert routing makes that possible.