Leakage in Machine Learning Pipelines

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

What leakage actually is

Data leakage happens when information that wouldn't be available at genuine prediction time ends up influencing the model during training or evaluation, making reported performance look better than what you'll actually get in production. It's insidious specifically because it doesn't cause errors or crashes — it causes silently inflated validation and test metrics that then fail to materialize once the model is deployed on truly fresh data, at which point debugging the gap is much harder than catching the leak upfront.

Preprocessing leakage

The most common form: fitting a preprocessing step (scaler, imputer, encoder, feature selector, PCA) on the full dataset — or on train+validation combined — before splitting, so statistics computed for preprocessing incorporate information from data the model will later be 'tested' on. The fix is mechanical but easy to forget under deadline pressure: every stateful preprocessing step must be fit only on the training fold, then applied (transform, not fit_transform) to validation and test data, and this must be repeated correctly inside every cross-validation fold, not done once globally before CV starts.

leakage_safe_pipeline.py

python

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression

# WRONG: fitting the scaler/selector before cross-validation leaks test-fold
# statistics into training for every fold.
#
# scaler = StandardScaler().fit(X)          # fit on ALL data -- leak
# X_scaled = scaler.transform(X)
# scores = cross_val_score(model, X_scaled, y, cv=5)

# RIGHT: wrap every stateful step in a Pipeline so each CV fold refits
# the scaler/selector using only that fold's training data.
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("select", SelectKBest(k=20)),
    ("model", LogisticRegression(max_iter=1000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, X, y, cv=cv)

Target leakage: features that encode the answer

Target leakage occurs when a feature is itself derived from, or a proxy for, the target variable in a way that wouldn't be available before the prediction needs to be made. A classic example: predicting whether a patient will be readmitted to a hospital, using a feature like 'discharge disposition includes hospice referral' — a variable that's effectively only set *because* severe outcomes occurred, making it nearly a restatement of the label rather than a genuine predictor available at prediction time. These leaks are harder to catch mechanically than preprocessing leakage because the code runs fine and the feature is a legitimate column in the dataset — catching it requires domain knowledge about the actual causal and temporal order of events.

Suspiciously high accuracy is the single best leakage signal

If a model achieves accuracy, AUC, or F1 that's unusually high for the difficulty of the task — especially a sudden jump after adding a new feature — the first hypothesis should be leakage, not model brilliance. Checking feature importance for a single feature that dominates all others, and manually reasoning about whether that feature could plausibly be known before the prediction is needed, catches a large fraction of real-world leakage bugs before they reach production.

Temporal leakage

Specific to time-dependent data: using future information to predict the past, usually introduced by a random (non-chronological) train/test split on time series data, or by features that are aggregated across a window that includes future timestamps relative to the prediction point (e.g. a 'total purchases this month' feature computed using the full month, when the prediction is meant to happen mid-month). The fix requires both a chronological split and careful feature-engineering discipline: every feature must be computed using only information available strictly before the prediction timestamp.

Leakage type	Typical cause	Primary fix
Preprocessing leakage	Fitting scalers/imputers/selectors on full data	Fit only on training fold, inside a Pipeline
Target leakage	Feature derived from or correlated with the label's cause	Domain review of feature provenance and timing
Temporal leakage	Features or splits using future information	Chronological split; point-in-time feature computation
Group leakage	Same entity's rows split across train/test	Group-aware splitting (GroupKFold)

Leakage is a pipeline property, not a single-step bug

Because leakage can enter at data collection, feature engineering, preprocessing, splitting, or evaluation, auditing for it requires tracing the full pipeline end to end, not just checking the modeling step in isolation. Building pipelines with scikit-learn's Pipeline/ColumnTransformer (or equivalent in other frameworks), rather than manually chaining fit/transform calls, makes the correct fit-on-train-only discipline structurally harder to violate by accident.

What's Next?

A clean, leakage-free pipeline is the foundation that makes every subsequent step — feature selection, model comparison, hyperparameter tuning — actually trustworthy; without it, even the most careful downstream methodology is built on a biased estimate.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →