Validation Curve Analysis
A different axis than the learning curve
A learning curve asks "how does error change as we add more training data, with the model fixed?" A validation curve asks a different question entirely: "how does error change as we vary one hyperparameter, with the training set size fixed?" Where learning curves diagnose whether you need more data, validation curves diagnose whether a specific hyperparameter — regularization strength, tree depth, number of neighbors, polynomial degree — is set too low or too high, directly visualizing the classic underfitting-to-overfitting transition as that single knob turns.
Constructing a validation curve
To build one: pick a hyperparameter and a range of candidate values, then for each value, train the model (typically with cross-validation) and record both training score and validation score. Plot the hyperparameter value on the x-axis and score (or error) on the y-axis, with one line for training and one for validation.
The x-axis is often log-scaled, since many hyperparameters that matter — regularization strength $\lambda$, learning rate, tree depth — have effects that are roughly multiplicative rather than additive across their useful range.
import numpy as np
from sklearn.model_selection import validation_curve
from sklearn.linear_model import Ridge
import matplotlib.pyplot as plt
alphas = np.logspace(-4, 4, 20)
train_scores, val_scores = validation_curve(
estimator=Ridge(),
X=X, y=y,
param_name="alpha",
param_range=alphas,
cv=5,
scoring="neg_mean_squared_error",
)
train_mean = -train_scores.mean(axis=1)
val_mean = -val_scores.mean(axis=1)
plt.semilogx(alphas, train_mean, "o-", label="Training error")
plt.semilogx(alphas, val_mean, "o-", label="Validation error")
plt.xlabel("alpha (regularization strength)")
plt.ylabel("Mean squared error")
plt.legend()
plt.title("Validation curve")
plt.show()
best_alpha = alphas[np.argmin(val_mean)]
print(f"Best alpha: {best_alpha:.4g}")The classic U-shape
A validation curve for complexity-controlling hyperparameters typically traces a U-shape in validation error (or an inverted-U in validation score) as complexity increases:
- Left side (too simple / too much regularization): both training and validation error are high and close together — the model is underfitting, too constrained to fit even the training data well.
- Middle (the sweet spot): validation error reaches its minimum. Training error is lower than validation error, but the gap is still modest.
- Right side (too complex / too little regularization): training error keeps dropping, often toward zero, while validation error rises back up — the model is overfitting, memorizing training-set idiosyncrasies that don't generalize.
The hyperparameter value at the minimum of the validation curve is the one you'd select — this is exactly what grid search and GridSearchCV automate at scale across one or more hyperparameters simultaneously.
Validation curves for depth, k, and degree
The same U-shape diagnostic applies to any complexity knob, not just explicit regularization:
- Decision tree `max_depth`: shallow trees underfit; very deep trees memorize individual training points, especially with noisy or small data.
- k-NN's `n_neighbors`: small $k$ (even $k=1$) overfits to local noise; large $k$ oversmooths and underfits by averaging over points that aren't actually similar.
- Polynomial regression degree: low degree underfits curved relationships; high degree oscillates wildly to pass near every training point (Runge's phenomenon is the extreme case).
- Neural network width/depth or dropout rate: more units or less dropout increases capacity and overfitting risk; too few units or too much dropout underfits.
In each case the x-axis variable is different but the shape of the diagnostic — and the reasoning for picking the minimum validation error — is identical.
Don't tune on the test set
Validation curves vs. grid search
GridSearchCV and RandomizedSearchCV extend the same cross-validation logic across the full joint hyperparameter space, at the cost of losing the simple 2D plot. In practice, many practitioners use a validation curve first to narrow a promising range for each hyperparameter, then run a grid or random search over the narrowed joint space.What's next?
Validation curves and learning curves together cover the two main axes of model diagnosis: data quantity and model complexity. With both tools in hand, the next step is usually combining them with a systematic hyperparameter search strategy and a clear evaluation protocol, so the choices these curves suggest get validated rigorously rather than cherry-picked from a single plot.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.