Ensemble Modeling: Bagging, Boosting & Stacking Compared

One family, three strategies

The idea that a committee of models beats any single model is one of the most consistently useful findings in machine learning. But 'combine multiple models' isn't a single technique — it's a family of strategies that solve different problems. The three major approaches are bagging (reduce variance by averaging independent models), boosting (reduce bias by sequentially correcting errors), and stacking (learn the optimal way to combine heterogeneous models). Understanding what each one actually fixes is the difference between using ensembles effectively and cargo-culting 'use an ensemble, it'll be better.'

Bagging: averaging away variance

Bagging (Bootstrap AGGregating) trains many copies of the same model type independently and in parallel, each on a different bootstrap sample (a random sample with replacement) of the training data. Predictions are then averaged (regression) or majority-voted (classification).

The statistical logic is straightforward: if you have several models that each make independent, unbiased errors, averaging their predictions cancels out much of the noise while the underlying signal reinforces itself. Bagging's entire benefit is variance reduction — it does nothing to fix a model that is systematically biased, since averaging several biased estimates just gives you a biased average.

Random Forest is bagging applied to decision trees, with an added twist: each tree also only considers a random subset of features at each split, which decorrelates the trees further and improves the variance reduction beyond what bootstrap sampling alone achieves.

Boosting: sequentially reducing bias

Boosting works the opposite way: models are trained sequentially, not independently, and each one is explicitly trained to fix the errors of the ensemble so far (as covered in the previous lesson on AdaBoost and gradient boosting). Because each model directly targets the current ensemble's shortcomings, boosting primarily attacks bias — it can turn a collection of individually weak models into a strong one that systematically captures patterns none of them could alone.

The cost is that boosting's sequential nature can't be parallelized across models the way bagging can, and because it keeps chasing residual errors, it carries a real risk of eventually overfitting if left unchecked — unlike bagging, which tends to plateau safely as you add more trees.

Stacking: learning how to combine different models

Stacking takes a different approach entirely: instead of many copies of the same model type, it combines several genuinely different model types (say, a logistic regression, a random forest, and a gradient boosted tree) and trains a meta-model to learn the best way to combine their predictions.

The procedure: train each base model on the training data (typically using cross-validation to generate out-of-fold predictions, to avoid leakage). Use those out-of-fold predictions as input features for a meta-model, often a simple linear or logistic regression. The meta-model learns, directly from data, how much to trust each base model's output in different situations — effectively learning a weighted vote that is smarter than a simple average.

Stacking tends to help most when the base models make genuinely different kinds of errors — a linear model and a tree-based model, for example, will tend to get different examples wrong, and the meta-model can learn to lean on whichever one is more reliable for a given kind of input.

Bagging vs. Boosting vs. Stacking

What's next

Ensembles combine supervised models to improve prediction accuracy. On the unsupervised side, Gaussian Mixture Models extend the clustering idea you saw in K-Means into a fully probabilistic framework that captures uncertainty in cluster assignment itself.