Learning Curves: Diagnosing Bias vs Variance

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The question every stuck model raises

When a model's performance plateaus below where you need it, there are two fundamentally different diagnoses, and they call for opposite fixes. High bias means the model is too simple to capture the underlying pattern — it underfits, performing poorly on both training and validation data. High variance means the model is fitting noise specific to the training set — it overfits, performing well on training data but poorly on validation data. Throwing more data at a high-bias problem wastes time; adding model capacity to a high-variance problem makes it worse. Learning curves are the standard diagnostic tool for telling these two situations apart before deciding what to do next.

What a learning curve actually plots

A learning curve plots model error (or score) on the y-axis against training set size on the x-axis, with two lines: training error and validation (or cross-validation) error. To generate one, you train the same model repeatedly on increasing fractions of the training data (e.g., 10%, 20%, ..., 100%), recording both training and validation error at each size.

This is different from a loss-vs-epoch training curve, which tracks the same model over optimization time. A learning curve tracks different amounts of *data*, holding the model and training procedure fixed, which is precisely what's needed to answer "would more data help?"

Plotting a learning curve with sklearn

python

import numpy as np
from sklearn.model_selection import learning_curve
import matplotlib.pyplot as plt

train_sizes, train_scores, val_scores = learning_curve(
    estimator=model,
    X=X, y=y,
    train_sizes=np.linspace(0.1, 1.0, 10),
    cv=5,
    scoring="neg_mean_squared_error",
)

train_mean = -train_scores.mean(axis=1)
val_mean = -val_scores.mean(axis=1)

plt.plot(train_sizes, train_mean, "o-", label="Training error")
plt.plot(train_sizes, val_mean, "o-", label="Validation error")
plt.xlabel("Training set size")
plt.ylabel("Error")
plt.legend()
plt.title("Learning curve")
plt.show()

Reading the high-bias (underfitting) signature

A high-bias model shows training error and validation error converging to a similarly high value, with a small gap between them, and the validation curve flattening early even as more training data is added. Both curves plateau at a disappointing error level because the model's hypothesis class simply lacks the expressive power to represent the true relationship, no matter how much data it sees. The fix isn't more data — it's a more expressive model (more features, higher-degree polynomial terms, a deeper network, less aggressive regularization), or better features that make the pattern easier to represent.

Reading the high-variance (overfitting) signature

A high-variance model shows a persistent, large gap between low training error and much higher validation error, where training error stays near zero while validation error stays elevated. Critically, as training set size increases, this gap tends to narrow — the validation curve keeps improving and the two curves are converging, just slowly, and often haven't met yet at the largest available training size.

This is the signature that tells you more data would likely help, because the trend line shows the gap closing. Alternatives that address the same underlying problem without needing more data include adding regularization (L1/L2 penalties, dropout), reducing model complexity, or using techniques like early stopping and data augmentation.

Bias vs. variance signatures on a learning curve

Signal	High bias (underfit)	High variance (overfit)
Training error	High, plateaus early	Low, near zero
Validation error	High, close to training error	High, well above training error
Gap between curves	Small	Large
Effect of more data	Little to no improvement	Gap narrows, validation error drops
Best fix	More capacity, better features, less regularization	More data, more regularization, simpler model

The bias-variance decomposition behind the diagram

Formally, expected test error decomposes as $\mathbb{E}[(\hat{f}(x) - f(x))^2] = \text{Bias}(\hat{f}(x))^2 + \text{Var}(\hat{f}(x)) + \sigma^2$, where $\sigma^2$ is irreducible noise. Bias is the error from wrong assumptions baked into the model (e.g., assuming linearity when the truth is curved); variance is the error from sensitivity to which specific training sample you happened to draw. Learning curves are an empirical, visual proxy for this decomposition — you can't measure bias and variance directly on real data without knowing the true function, but the gap-and-plateau pattern across training sizes gives you a reliable practical substitute.

What's next?

Learning curves vary training set size while holding model complexity and hyperparameters fixed. The complementary diagnostic — holding training set size fixed and varying a hyperparameter like regularization strength or tree depth — answers a closely related but distinct question: not "would more data help?" but "is this specific hyperparameter setting too simple or too complex?"

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →