Random Forest
The wisdom of crowds, applied to trees
Decision trees have a fatal flaw: they're high-variance. Small changes in training data produce wildly different trees. A single tree's prediction can't be trusted.
The solution: don't use one tree. Train hundreds, let each one vote, take the majority. Individual errors cancel out. The aggregate is stable.
This is ensemble learning — the idea that a collection of weak, somewhat independent predictors beats any single strong predictor. The evidence for this has been overwhelming since the 1990s, and Random Forest remains one of the top-performing algorithms on tabular data to this day.
Two sources of randomness
Random Forest gets its randomness — and its power — from two places:
Bagging (bootstrap aggregating): Each tree is trained on a different random sample of the training data, drawn with replacement. This means each tree sees roughly 63% of the original data (different rows), while ~37% is held out. Trees trained on different data make different mistakes.
Random feature selection: At every split in every tree, only a random subset of features is considered — typically √n_features for classification, n_features/3 for regression. This forces trees to be different even when they're seeing similar data, and stops any single dominant feature from appearing at the root of every tree.
Note
Feature importance: a free gift
Random Forest gives you something most models don't: a reliable ranking of which features matter.
Every time a feature is used to make a split across all trees, we can measure how much it reduced impurity on average. Features used more often, earlier in trees, to create purer splits get higher importance scores.
This is useful for: identifying which columns in a dataset to invest in, dropping irrelevant features to speed up training, and explaining model decisions to stakeholders. It's not a perfect measure (correlated features can split importance between them), but it's the best free feature importance you'll get from any algorithm.
Out-of-bag error: free validation
Because each tree only sees ~63% of the training data (due to bootstrap sampling), the remaining ~37% can act as a validation set for that tree. Average this across all trees, and you get the out-of-bag (OOB) error — a reliable estimate of generalisation error without ever splitting off a separate validation set.
Set oob_score=True in sklearn and you get this for free. On small datasets, this is especially valuable — you don't waste any data on a validation split.
Key hyperparameters
n_estimators — the number of trees. More is always at least as good, just slower. 100–500 covers most use cases. Returns diminish quickly after 200.max_depth — maximum depth per tree. None (default) lets trees grow fully. Limiting depth trades some accuracy for speed and slightly lower variance.max_features — features considered per split. Default sqrt(n_features) for classification. Lower values create more diverse trees.min_samples_leaf — minimum samples required at a leaf. Higher values smooth predictions and reduce overfitting on small datasets.
Note
What's next
Random Forest handles classification and regression. K-Means Clustering is a different kind of ML — unsupervised, meaning you don't have labels at all. The goal isn't to predict a known output but to discover hidden structure in the data.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.