Cross-Validation Strategies

Why a single train/test split isn't enough

Splitting your data once into train and test sets gives you exactly one estimate of how well a model generalizes. That estimate has variance — a different random split could give a noticeably different score, especially with smaller datasets. Cross-validation addresses this by repeating the train/test split multiple times in a structured way and averaging the results, giving a more stable and trustworthy estimate of model performance.

But the structure of the split matters enormously. The right scheme depends entirely on how your data is generated — independent rows, grouped rows, imbalanced classes, or a time-ordered sequence all demand different splitting logic. Using the wrong one doesn't just give a slightly noisier estimate; it can give a confidently wrong one.

K-fold cross-validation: the default

K-fold cross-validation splits the dataset into $k$ equal-sized folds. For each of $k$ rounds, one fold is held out as the validation set and the model trains on the remaining $k-1$ folds. The final score is the average across all $k$ rounds. With $k=5$, every data point is used for validation exactly once and for training four times.

This is the right default when rows are genuinely independent and identically distributed — tabular data without time structure, without grouping, and without severe class imbalance. It uses data efficiently (unlike a single hold-out split, every row contributes to both training and validation across the folds) and gives a variance estimate across the $k$ scores, not just a single number.

Stratified k-fold: protecting class balance

Plain k-fold shuffles rows randomly into folds, which can produce folds with wildly different class distributions when classes are imbalanced. If 5% of your data is the positive class, a random fold might contain only 1% positives by chance, making both training and evaluation unreliable in that fold.

Stratified k-fold fixes this by preserving the overall class proportion in every fold. Each fold gets approximately the same percentage of each class as the full dataset. This is the standard choice for classification problems, and doubly important for imbalanced classification (fraud detection, rare disease diagnosis, churn), where a single unlucky fold split can otherwise produce a fold with zero minority-class examples.

Leave-one-out cross-validation (LOOCV)

Leave-one-out cross-validation is the extreme case of k-fold where $k = n$ — the number of samples. Each round trains on all but one data point and validates on that single point, repeating $n$ times. It uses the maximum possible amount of training data in every round and produces an almost unbiased estimate of generalization error.

The cost is computational: training $n$ separate models is prohibitively expensive for anything beyond small datasets, and the resulting estimate tends to have high variance across different datasets (removing or adding a single point can meaningfully change which model wins). LOOCV is mainly useful when you have a small dataset — tens to low hundreds of samples — where ordinary k-fold would leave each fold too small to be a meaningful test set.

Time series: walk-forward validation

None of the above are valid when rows have a time order, because standard k-fold would train on future data and validate on the past — a form of leakage that makes the model look far better than it will in production. For time-ordered data, use walk-forward (expanding window) validation: train on data up to time $t$, validate on the next chunk, then slide the boundary forward and repeat, always keeping training data strictly before validation data in time.

There are two common variants: the expanding window, where the training set grows with each step (keeping all past data), and the rolling window, where training uses a fixed-size moving window (useful when old data becomes stale, e.g., in financial markets with regime changes). Either way, the fold boundaries respect temporal order — no fold ever validates on data that precedes its training data.

Choosing a cross-validation strategy

What's next

Cross-validation gives you a reliable performance estimate for one model — but comparing two models' cross-validation scores and declaring a winner needs more care than just comparing two averages. The next lesson on statistical hypothesis testing for ML model comparison covers how to determine whether an observed difference is real or just noise.