Time Series Cross-Validation

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Why shuffled k-fold breaks on time series

Standard k-fold cross-validation assumes observations are independent and interchangeable, so it shuffles data freely across folds. Time series data violates that assumption: each observation is correlated with its neighbors, and the whole point of a forecasting model is to predict the future from the past. If you randomly shuffle a time series into folds, a model can end up training on data from *after* the point it's being tested on — it learns to 'predict' the past using information from the future, a form of leakage that makes validation scores look far better than real deployment performance will be.

Time series cross-validation respects the arrow of time: every training set only contains observations that occurred before the corresponding validation set.

Walk-forward validation (expanding window)

The standard approach is walk-forward validation, also called rolling-origin evaluation. You pick a sequence of cutoff points in time. At each cutoff, the model trains on all data up to that point and is evaluated on a block of data immediately after it. The cutoff then advances, the training window grows (or slides), and the process repeats.

Two common window styles:

Expanding window: training set always starts at $t=0$ and grows with each split — more historical data each time.

Sliding window: training set has a fixed size and slides forward, dropping old data as new data is added — useful when older data is less relevant (e.g. concept drift).

Expanding vs. sliding window splits

text

Expanding window:
  Split 1: [TRAIN=====]            [TEST]
  Split 2: [TRAIN==========]            [TEST]
  Split 3: [TRAIN===============]            [TEST]

Sliding window (fixed train size):
  Split 1: [TRAIN=====]            [TEST]
  Split 2:      [TRAIN=====]            [TEST]
  Split 3:           [TRAIN=====]            [TEST]

In both: TEST always follows TRAIN in time. No test block ever precedes
or overlaps the data used to train the model evaluated on it.

Walk-forward CV with sklearn's TimeSeriesSplit

python

import numpy as np
from sklearn.model_selection import TimeSeriesSplit
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

# X, y are ordered chronologically — never shuffle a time series
tscv = TimeSeriesSplit(n_splits=5, test_size=30, gap=0)

scores = []
for fold, (train_idx, test_idx) in enumerate(tscv.split(X)):
    model = RandomForestRegressor(n_estimators=200, random_state=0)
    model.fit(X[train_idx], y[train_idx])
    preds = model.predict(X[test_idx])
    mae = mean_absolute_error(y[test_idx], preds)
    scores.append(mae)
    print(f"Fold {fold}: train={len(train_idx)} test={len(test_idx)} MAE={mae:.3f}")

print(f"Mean MAE across folds: {np.mean(scores):.3f}")

Add a gap to simulate real forecast lead time

In production, a forecast is rarely consumed instantly — there's usually a lag between when data becomes available and when the forecast is needed (e.g. you forecast tomorrow's demand using data only up to yesterday evening). TimeSeriesSplit's gap parameter lets you insert a buffer between the end of the training window and the start of the test window, so your validation mirrors the actual lead time your model will face in deployment, rather than giving it an unrealistically fresh training cutoff.

Blocked cross-validation and leakage from features

A subtler leakage source: even with correctly ordered splits, feature engineering steps like rolling averages, scaling, or imputation can leak if computed on the full dataset *before* splitting (e.g. a rolling mean computed across the entire series, or a StandardScaler fit on train+test combined). The fix is the same discipline as with nested CV — fit any preprocessing exclusively on the training portion of each fold, then apply it to that fold's test portion. Blocked CV is a related variant that additionally removes a buffer of points directly adjacent to the split boundary, guarding against autocorrelation letting information bleed across the train/test edge even when ordering is respected.

Don't average metrics across folds without checking variance

Time series fold-to-fold performance can vary wildly if the series has regime changes (a pandemic, a policy change, a seasonal spike). A single averaged MAE across folds can hide a model that's great in calm periods and catastrophic during volatility. Always inspect per-fold scores, not just the mean, before trusting a walk-forward evaluation.

What's Next?

Walk-forward validation tells you how well a model generalizes forward in time, but it says nothing about whether the series itself is well-behaved for classical forecasting methods — many of which assume the statistical properties of the series don't change over time. That assumption, and how to test for it, is the next thing worth understanding before fitting a forecasting model.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →