Voting Classifiers (Hard vs Soft)
The simplest ensemble: just vote
A voting classifier combines predictions from several independently trained models with no learned combination step at all — it's the simplest possible ensemble. Despite that simplicity, it reliably improves over any single constituent model when the base models are reasonably accurate and make uncorrelated errors, which is the same statistical principle behind bagging and random forests: averaging independent, unbiased estimators reduces variance.
Hard voting: majority rule on labels
Hard voting collects each base classifier's predicted class label and outputs whichever label received the most votes (plurality, or a strict majority for binary problems). It only needs discrete predictions, not probabilities, so it works even with classifiers that don't produce calibrated confidence scores. The downside: it throws away information about *how confident* each model was. A model that's 51% sure gets exactly the same single vote as a model that's 99% sure.
Soft voting: averaging probabilities
Soft voting instead averages the predicted class probabilities across all base models and picks the class with the highest average probability: $P(y=k) = \frac{1}{M}\sum_{m=1}^{M} P_m(y=k)$. This preserves confidence information and generally outperforms hard voting *when the base models' probability estimates are reasonably well-calibrated*. If one model's probabilities are badly miscalibrated (e.g., an SVM without proper Platt scaling), soft voting can actually do worse than hard voting, because the miscalibrated model's confident-but-wrong predictions dominate the average.
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_wine
X, y = load_wine(return_X_y=True)
estimators = [
("lr", LogisticRegression(max_iter=1000)),
("nb", GaussianNB()),
("dt", DecisionTreeClassifier(max_depth=5)),
]
hard = VotingClassifier(estimators=estimators, voting="hard")
soft = VotingClassifier(estimators=estimators, voting="soft")
print("Hard voting:", cross_val_score(hard, X, y, cv=5).mean())
print("Soft voting:", cross_val_score(soft, X, y, cv=5).mean())
for name, model in estimators:
print(f"{name} alone:", cross_val_score(model, X, y, cv=5).mean())
Weighting votes is tempting but easy to overfit
| Aspect | Hard voting | Soft voting |
|---|---|---|
| Input needed | Predicted labels only | Predicted probabilities |
| Uses confidence info | No | Yes |
| Sensitive to miscalibration | No | Yes |
| Typical performance | Good baseline | Better, if models are calibrated |
What's next?
Voting ensembles combine independently trained models after the fact. Bagged ensembles like random forests instead build that diversity in from the start, by training many models on resampled subsets of the data — which opens up a different, essentially free way to estimate generalization error without a separate validation set.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.