Decision Stumps & Weak Learners
The simplest classifier that still works
A decision stump is a decision tree with exactly one split. It picks a single feature, picks a single threshold, and classifies every point based purely on which side of that threshold it falls. That's the entire model — no further branching, no interactions between features, nothing.
On its own, a decision stump is almost always a bad classifier. Real data rarely separates cleanly along one feature with one threshold. But this weakness is exactly what makes decision stumps useful: they're the textbook example of a weak learner — a model that performs only slightly better than random guessing — and weak learners are the fundamental building block of boosting algorithms.
Fitting a stump: brute-force search over splits
Fitting a decision stump is simple enough to do by brute force: for every feature, sort the values, consider every possible threshold between consecutive values, and compute how well that split separates the classes (using a criterion like Gini impurity or information gain, the same criteria used at every node of a full decision tree). Keep the single best (feature, threshold) pair found across all features.
Because there's no recursion — no second split to consider — this search is fast, which is part of why stumps are cheap to fit repeatedly inside a boosting loop.
Note
Why AdaBoost uses stumps specifically
AdaBoost is the clearest illustration of why weak learners matter. It fits a sequence of weak learners (stumps are the default and most common choice), where each new stump is trained on a reweighted version of the data — points misclassified by previous stumps get higher weight, forcing the next stump to focus on exactly the examples the ensemble is currently getting wrong. Each stump's vote in the final prediction is also weighted by how accurate it was.
Using a model more complex than a stump defeats the purpose: a single strong learner (like a deep decision tree) can already fit the training data well, so there's little left for subsequent rounds to correct, and the ensemble gains little beyond what one strong model would already achieve. Stumps keep each round intentionally weak, so boosting has somewhere meaningful to go over many rounds.
Decision stump vs. full decision tree as a base learner
| Dimension | Decision stump (depth 1) | Full decision tree |
|---|---|---|
| Bias | High — captures only one feature's effect | Low — can fit training data closely |
| Variance | Low — very stable across resamples | Higher — sensitive to training data changes |
| Fit speed | Very fast — one split, brute-force search | Slower — recursive splitting |
| Role in boosting | Ideal weak learner — leaves room to improve each round | Risks overfitting early, limiting boosting's benefit |
| Role in bagging (e.g. random forest) | Too weak alone — needs many features and depth to decorrelate | Standard choice — depth captures interactions |
Note
What's next
Decision stumps show how simple components can combine into something powerful through the right training procedure. Hopfield Networks, covered next, show an even older and more surprising version of this idea — simple units whose collective dynamics let a network "remember" entire patterns.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.