Model Stacking in Practice
Stacking: a model that learns to combine models
Stacking (stacked generalization, introduced by Wolpert in 1992) trains several diverse base models, then trains a second-level meta-learner whose input features are the base models' own predictions. Instead of a fixed combination rule like averaging or majority vote, the meta-learner learns the optimal way to weight and combine the base models' outputs, including learning that model A is more trustworthy on some regions of the input space than model B.
Why you can't just train the meta-learner on in-sample predictions
If the meta-learner is trained on predictions the base models made on the *same data they were trained on*, it will see artificially confident, overfit predictions and learn to over-trust them — a severe form of leakage. The standard fix is out-of-fold prediction: split training data into $K$ folds, train each base model on $K-1$ folds and predict on the held-out fold, repeating so every training row gets an out-of-fold prediction from every base model. The meta-learner trains on these honest, held-out predictions. At inference time, base models (now retrained on the full training set) generate predictions for new data, and the meta-learner combines them.
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
base_learners = [
("svc", SVC(probability=True, kernel="rbf")),
("tree", DecisionTreeClassifier(max_depth=4)),
]
# StackingClassifier handles out-of-fold prediction internally via cv=5
stack = StackingClassifier(
estimators=base_learners,
final_estimator=LogisticRegression(),
cv=5,
stack_method="predict_proba",
)
print("Stacked accuracy:", cross_val_score(stack, X, y, cv=5).mean())
for name, model in base_learners:
print(f"{name} alone:", cross_val_score(model, X, y, cv=5).mean())
Diversity matters more than individual base-model strength
Choosing the meta-learner
A simple, low-variance model — typically logistic/linear regression, or a shallow tree — is the usual choice for the meta-learner, precisely because it's being trained on a small number of highly informative features (the base models' predictions) and complex meta-learners risk overfitting *that* layer instead. In Kaggle-style competition stacking, it's common to even stack multiple levels (base models → first meta-layer → second meta-layer), but each additional layer compounds both diminishing returns and leakage risk, so production systems rarely go beyond two levels.
| Ensembling method | Combination rule | Learns per-input weighting? |
|---|---|---|
| Averaging / voting | Fixed (mean or majority) | No |
| Weighted averaging | Fixed weights, tuned once | No |
| Stacking | Learned by meta-model | Yes |
What's next?
Stacking is the most flexible but also the most leakage-prone ensembling approach. Simpler combination rules — straightforward voting across classifiers — trade away that flexibility for something much easier to implement and reason about correctly, which is worth understanding before reaching for a meta-learner.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.