Chinchilla & Compute-Optimal Training

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The question Chinchilla set out to answer

Given a fixed training compute budget $C$ (measured in FLOPs), how should it be split between model size $N$ (parameters) and dataset size $D$ (tokens) to minimize final training loss? DeepMind's 2022 Chinchilla paper (Hoffmann et al.) answered this with a large, carefully controlled sweep: training over 400 models ranging from 70 million to 16 billion parameters on 5 to 500 billion tokens, then fitting the resulting loss surface as a function of both $N$ and $D$ jointly, rather than studying each in isolation the way earlier scaling law work had.

The headline result: scale both, roughly together

Chinchilla's central finding was that model size and training tokens should scale at roughly the same rate as compute budget grows — for every doubling of compute, both $N$ and $D$ should roughly double too, rather than compute being funneled disproportionately into parameter count as earlier guidance (and most real training runs of the time) had done. The paper's own named model, Chinchilla (70B parameters, trained on 1.4 trillion tokens), was deliberately built smaller and trained on more data than its predecessor Gopher (280B parameters, trained on far fewer tokens) at a comparable training compute budget — and it outperformed Gopher on a broad suite of downstream benchmarks, demonstrating the claim empirically rather than just theoretically.

chinchilla_optimal_split.py

python

# Approximate Chinchilla-style compute-optimal split
# (illustrative constants based on the paper's fitted exponents)

def chinchilla_optimal(compute_flops):
    # N_opt and D_opt both scale roughly as C^0.5 under Chinchilla's fit
    # (exact exponents and constants are empirically fitted, shown schematically here)
    a, b = 0.49, 0.51              # close to a roughly equal split
    N_opt = 0.6 * compute_flops ** a
    D_opt = 0.3 * compute_flops ** b
    return N_opt, D_opt

for compute in [1e21, 1e22, 1e23, 1e24]:
    N_opt, D_opt = chinchilla_optimal(compute)
    print(f"Compute={compute:.0e} FLOPs -> N~{N_opt:.2e} params, D~{D_opt:.2e} tokens, "
          f"ratio D/N~{D_opt/N_opt:.1f}")

The ~20 tokens-per-parameter rule of thumb

A frequently cited practical takeaway from Chinchilla's fitted relationship is a rough rule of thumb: compute-optimal training uses roughly 20 tokens of training data per model parameter (i.e., $D \approx 20N$). This number should be treated as an approximation derived from the specific data distribution and architecture Chinchilla studied, not a universal law — later work has shown the optimal ratio shifts with data quality, architecture details, and especially with the intended *inference* budget (see below), but it remains a widely used starting heuristic for planning a new training run's rough scale.

Gopher vs. Chinchilla, at comparable training compute

Model	Parameters	Training tokens	Downstream result
Gopher	280B	~300B	Baseline, larger model, relatively undertrained
Chinchilla	70B (4x smaller)	~1.4T (4-5x more)	Outperformed Gopher on the majority of evaluated benchmarks

Chinchilla-optimal isn't the same as inference-optimal

Chinchilla's recipe minimizes *training* loss for a fixed *training* compute budget — it says nothing about inference cost. A smaller model trained on proportionally more tokens than Chinchilla-optimal (sometimes called 'overtraining' relative to Chinchilla) can still be the better overall choice if the model will be served at massive inference volume, since a smaller model costs less per inference call for the rest of its deployed lifetime. Several widely deployed open models have deliberately trained smaller models on far more than the Chinchilla-optimal token count for exactly this reason.

Data quality and availability complicate the clean ratio

Chinchilla's fitted ratio assumed a particular data distribution and quality level; it doesn't address what happens when high-quality training data of the needed volume simply isn't available at the token counts larger compute budgets would call for — a real and growing constraint as frontier training runs approach or exceed the scale of easily available high-quality public text data.

What's next?

Scaling laws and compute-optimal training describe smooth, predictable improvements in loss — but some of the most consequential changes in model behavior as scale increases are reported as anything but smooth, which is exactly the puzzle at the center of the debate over emergent abilities in large language models.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →