Sampling Strategies: Temperature, Top-k & Top-p
From logits to a chosen token
At every generation step, an LLM outputs a probability distribution over its entire vocabulary — tens of thousands of possible next tokens, each with a probability derived from the model's raw logits via softmax: $P(x_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$. Deciding which token to actually emit from this distribution is called decoding, and the choice of sampling strategy has a huge effect on output quality, creativity, and reliability — independent of the model's underlying capability.
Greedy decoding and temperature
The simplest strategy, greedy decoding, always picks the single highest-probability token. It's deterministic and often produces repetitive, bland text because it never explores lower-probability-but-reasonable alternatives. Temperature ($T$) reshapes the distribution before sampling by dividing logits by $T$ before softmax: $P(x_i) = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}$. $T < 1$ sharpens the distribution (more deterministic, closer to greedy), $T > 1$ flattens it (more random, more diverse), and $T = 1$ leaves it unchanged. Temperature alone doesn't remove any tokens from consideration — it only rescales how peaked or flat the distribution is.
Top-k: truncate to the k most likely tokens
Top-k sampling keeps only the $k$ tokens with the highest probability, renormalizes their probabilities to sum to 1, and samples from that truncated set. This prevents the model from ever picking an absurdly unlikely token, no matter how long the tail of the distribution is. The downside is that $k$ is fixed regardless of context: in a very confident distribution (one token at 95% probability), top-k with $k=50$ still includes 49 essentially irrelevant tokens; in a very uncertain distribution, a fixed $k$ might exclude reasonable candidates.
Top-p (nucleus sampling): truncate by cumulative probability
Top-p sampling, also called nucleus sampling (Holtzman et al., 2019), fixes top-k's rigidity by choosing the smallest set of tokens whose cumulative probability exceeds a threshold $p$ (e.g. 0.9), then sampling from that set. When the model is confident, the nucleus might contain just 1-2 tokens; when it's uncertain, it naturally expands to include more candidates. This adaptivity is why top-p (often combined with a modest temperature) is the default decoding strategy in most production chat systems.
import numpy as np
def softmax(logits, temperature=1.0):
z = np.array(logits) / temperature
z = z - z.max()
exp_z = np.exp(z)
return exp_z / exp_z.sum()
def top_k_sample(probs, k, rng):
idx = np.argsort(probs)[::-1][:k]
p = probs[idx] / probs[idx].sum()
return rng.choice(idx, p=p)
def top_p_sample(probs, p, rng):
order = np.argsort(probs)[::-1]
sorted_probs = probs[order]
cumsum = np.cumsum(sorted_probs)
cutoff = np.searchsorted(cumsum, p) + 1
nucleus = order[:cutoff]
renorm = probs[nucleus] / probs[nucleus].sum()
return rng.choice(nucleus, p=renorm)
rng = np.random.default_rng(0)
logits = [2.1, 1.8, 0.4, -1.0, 3.0, 0.1]
probs = softmax(logits, temperature=0.8)
print("top-k=3 pick:", top_k_sample(probs, k=3, rng=rng))
print("top-p=0.9 pick:", top_p_sample(probs, p=0.9, rng=rng))Decoding strategies compared
| Strategy | Candidate set size | Adapts to model confidence? | Common use |
|---|---|---|---|
| Greedy | 1 (argmax) | No | Deterministic tasks, debugging |
| Temperature only | Full vocabulary, reshaped | No | Rarely used alone |
| Top-k | Fixed k | No | Simple, predictable diversity control |
| Top-p (nucleus) | Variable, by cumulative mass | Yes | Default for most chat/completion APIs |
Low temperature doesn't mean 'more correct'
What's Next?
These strategies all sample a single path through the token tree one step at a time. For tasks where you want to compare multiple possible continuations before committing, decoding can be extended into a genuine search problem over sequences rather than a greedy or random walk.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.