Nested Cross-Validation

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Why plain cross-validation can lie to you

Ordinary k-fold cross-validation is used for two different jobs at once: picking the best hyperparameters, and estimating how well the final model will generalize. The problem is that when you use the *same* CV splits to both select a hyperparameter (say, picking the best $C$ for an SVM via grid search) and report the resulting score, that score is optimistically biased. You've effectively let the test folds leak into the selection process — the hyperparameter was chosen precisely because it scored well on those folds, so the reported number overstates real-world performance.

Nested cross-validation fixes this by using two loops: an inner loop for hyperparameter selection, and an outer loop that evaluates the selected model on data it has never influenced.

The two-loop structure

Nested CV has an outer loop and an inner loop:

Outer loop (e.g. 5-fold): splits the data into outer-train and outer-test. The outer-test fold is held out completely — it is never touched during model selection.

Inner loop (e.g. 3-fold or more), run *inside* each outer-train fold: performs the hyperparameter search (grid search, random search, Bayesian optimization) using only outer-train data, cross-validating within it to pick the best hyperparameters.

Once the inner loop picks the best hyperparameters for that outer split, a model is retrained on the full outer-train set with those hyperparameters and scored on the outer-test fold. This repeats for every outer fold. The final performance estimate is the average of the outer-test scores — each one computed on data the selection process never saw.

Nested CV structure (5 outer x 3 inner)

text

Outer fold 1: [TEST ][-------------- TRAIN --------------]
                        Inner fold a: [train][train][val]
                        Inner fold b: [train][val][train]
                        Inner fold c: [val][train][train]
              -> pick best hyperparams from inner average
              -> retrain on full outer-train with best hyperparams
              -> score once on outer TEST fold  <-- unbiased estimate

Outer fold 2: [-------------- TRAIN --------------][TEST ]
              ... repeat ...

Final estimate = mean(outer_test_scores across all 5 outer folds)

Nested CV with scikit-learn

python

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold

X, y = load_breast_cancer(return_X_y=True)

param_grid = {"C": [0.1, 1, 10, 100], "gamma": [0.001, 0.01, 0.1]}

inner_cv = KFold(n_splits=3, shuffle=True, random_state=0)
outer_cv = KFold(n_splits=5, shuffle=True, random_state=1)

# Inner loop: GridSearchCV handles hyperparameter selection
clf = GridSearchCV(estimator=SVC(), param_grid=param_grid, cv=inner_cv)

# Outer loop: cross_val_score refits clf (full inner search) per outer fold
nested_scores = cross_val_score(clf, X, y, cv=outer_cv)

print("Outer fold scores:", np.round(nested_scores, 3))
print(f"Unbiased estimate: {nested_scores.mean():.3f} +/- {nested_scores.std():.3f}")

A single train/test split after tuning isn't enough

It's tempting to tune hyperparameters with k-fold CV, pick the winner, then evaluate once on a held-out test set and call it unbiased. That single split is fine, but with a small test set the estimate carries high variance — you don't know how much of the score is signal versus noise from that one particular split. Nested CV averages over many outer test folds, giving both an unbiased estimate *and* a measure of its variance across folds.

Cost, and when it's worth paying

Nested CV is expensive: with 5 outer folds and a 3-fold inner search over $g$ hyperparameter combinations, you train roughly $5 \times 3 \times g$ models, plus 5 final refits. For a grid of 20 combinations that's over 300 model fits for a single evaluation. This is why nested CV is primarily a tool for *comparing algorithms* or *reporting a trustworthy generalization estimate* in a paper or a high-stakes deployment decision — not something you run every time you tweak a model during exploratory work. In practice, many teams use plain CV (or a single validation split) for day-to-day tuning, and reserve nested CV for the final, reportable number.

Nested CV vs. single-split CV vs. plain k-fold CV

Approach	Hyperparameter selection	Generalization estimate	Cost
Plain k-fold CV	Yes, using CV score	Biased (optimistic)	Low
Train/val/test split	Yes, using val score	Unbiased, high variance	Low
Nested CV	Yes, inner loop only	Unbiased, lower variance	High

What's Next?

Nested CV assumes your data points are exchangeable — that shuffling them into folds doesn't destroy meaningful structure. That assumption breaks down for data with a time axis, where the future must never leak into the past during validation. The next natural question is how cross-validation needs to change when observations are ordered in time rather than independent.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →