Scaling Laws for Neural Networks
Predictable power laws in an otherwise unpredictable field
Deep learning is notoriously empirical — architectures and training recipes are often validated by trial and error rather than derived from first principles. Scaling laws are a notable exception: across a wide range of model families, researchers (most influentially Kaplan et al. at OpenAI in 2020) found that test loss follows remarkably smooth, predictable power-law relationships with model size, dataset size, and compute, holding across many orders of magnitude. This predictability is what let labs plan multi-million-dollar training runs with reasonable confidence in the resulting model's performance before ever running the full-scale training job.
The basic power-law form
The Kaplan et al. scaling laws expressed test loss $L$ as a power-law function of each resource independently (holding others non-limiting): $L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}$ for parameter count $N$, and similarly for dataset size $D$ and compute $C$, each with its own empirically fit exponent $\alpha$ and constant. The practical upshot: plotted on log-log axes, loss decreases as a straight line as you scale a resource, which is both a strikingly clean empirical regularity and directly useful for extrapolation — fit the law on affordable small-scale runs, then predict performance at a much larger, more expensive scale before committing the compute budget to actually running it.
import numpy as np
from scipy.optimize import curve_fit
def power_law_loss(N, N_c, alpha):
return (N_c / N) ** alpha
# Example: measured validation loss at several smaller model sizes
param_counts = np.array([1e6, 1e7, 1e8, 1e9])
measured_loss = np.array([4.2, 3.1, 2.4, 1.9])
popt, _ = curve_fit(power_law_loss, param_counts, measured_loss, p0=[1e5, 0.1])
N_c, alpha = popt
# Extrapolate to a much larger, not-yet-trained model size
predicted_loss_70b = power_law_loss(70e9, N_c, alpha)
print(f"Predicted loss at 70B params: {predicted_loss_70b:.3f}")Chinchilla's revision: compute-optimal allocation
The original Kaplan-era scaling laws implicitly suggested that, for a fixed compute budget, it was better to make models much bigger and train them on relatively less data. DeepMind's Chinchilla paper (Hoffmann et al., 2022) re-derived the scaling relationships more carefully and found this was a miscalibration — for a fixed compute budget, loss is minimized by scaling model size and training data roughly in tandem, not by over-indexing on parameter count alone. This single correction changed how most labs allocated training compute going forward, generally producing smaller, more heavily-trained models that matched or beat larger, more lightly-trained predecessors at the same compute cost.
Kaplan (2020) vs. Chinchilla (2022) scaling guidance
| Aspect | Kaplan et al. (2020) | Chinchilla / Hoffmann et al. (2022) |
|---|---|---|
| Compute-optimal strategy | Favor larger models, relatively less data per parameter | Scale model size and data roughly proportionally |
| Implication for a fixed budget | Train a very large model, somewhat undertrained | Train a smaller model, more fully to convergence on more data |
| Evidence base | Smaller-scale experiments, some methodological gaps later identified | Larger, more carefully controlled sweep of model/data/compute combinations |
| Practical effect on the field | Motivated the initial race toward ever-larger models | Shifted many subsequent models toward better compute-to-performance ratios |
Scaling laws are predictive, not explanatory
Extrapolation has limits
What's next?
Chinchilla's specific compute-optimal recipe is worth examining in more detail on its own, since the exact token-to-parameter ratio it recommends has become one of the most frequently cited and applied numbers in large-scale model planning.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.