Train/Validation/Test Split Strategies
Three sets, three distinct jobs
The training set is what the model actually learns its parameters from. The validation set is used during development to compare models, tune hyperparameters, and decide when to stop training — it influences your choices, so performance on it is an optimistic (biased) estimate of true generalization. The test set is touched exactly once, at the very end, to produce an unbiased estimate of how the final chosen model performs on genuinely unseen data. Conflating these roles — tuning hyperparameters against what you call your 'test set,' or reusing a validation set across dozens of experiments without accounting for the resulting selection bias — is one of the most common sources of inflated, unreliable reported performance in applied ML.
Typical split ratios and when they shift
A common starting point is 70/15/15 or 80/10/10 for train/validation/test, but the right ratio depends heavily on dataset size. With very large datasets (millions of examples), even 1% held out for validation and test can be more than enough absolute examples to get a tight, low-variance performance estimate, freeing up more data for training. With small datasets (hundreds to low thousands of examples), a fixed-ratio split can leave the validation or test set too small to produce a stable estimate, which is exactly the regime where cross-validation becomes more valuable than a single static split.
from sklearn.model_selection import train_test_split
# Two-step split: first carve out test, then split remainder into train/val
X_train_full, X_test, y_train_full, y_test = train_test_split(
X, y, test_size=0.15, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
X_train_full, y_train_full, test_size=0.1765, # ~15% of original total
stratify=y_train_full, random_state=42
)
# For time series: NEVER shuffle. Split chronologically.
split_date = "2024-01-01"
train_mask = df["date"] < split_date
test_mask = df["date"] >= split_date
X_train, X_test = X[train_mask], X[test_mask]When a single split isn't enough: k-fold cross-validation
A single train/validation split gives one noisy estimate of validation performance — different random splits can give meaningfully different numbers, especially with small data. K-fold cross-validation splits the training data into $k$ folds, trains $k$ separate models (each holding out a different fold for validation), and averages the resulting metric, trading more compute for a lower-variance, more reliable estimate. The test set is still held out separately and untouched throughout this process — cross-validation replaces repeated train/validation splitting, not the final test evaluation.
Random splitting is wrong for time-dependent or grouped data
| Strategy | Handles time dependence? | Handles grouped entities? | Variance of estimate |
|---|---|---|---|
| Single random split | No | No | High (small data), low (large data) |
| Stratified split | No | No | Lower than plain random, for class balance |
| K-fold cross-validation | No (unless time-aware variant used) | No (unless group-aware variant used) | Low (averaged across folds) |
| Chronological / rolling split | Yes | No | Depends on window size |
| GroupKFold | No | Yes | Low, with group integrity preserved |
The test set's whole value is in never being touched
What's Next?
Even a perfectly designed split can't save a pipeline where information from outside the training set has already leaked in upstream — which is why leakage deserves to be understood as its own distinct failure mode, not just a split-ratio problem.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.