Decision Trees

A model you can read

A decision tree is exactly what it sounds like: a tree of decisions. Each internal node asks a yes/no question about a feature. Each branch represents an answer. Each leaf is a final prediction.

A doctor diagnosing pneumonia might ask: "Is the patient's temperature above 38°C?" → yes → "Is the respiratory rate above 20?" → yes → "Probable pneumonia." A decision tree learns this kind of logic automatically from data.

The appeal is interpretability. Unlike a neural network with millions of weights, you can print a decision tree and follow its logic by hand. In regulated industries — credit, medicine, insurance — this matters enormously.

How the tree is built

Building a decision tree is a greedy, recursive process:

1. Start with all training examples at the root
2. For each possible feature and split point, measure how much purity improves if you split there
3. Pick the split that gives the greatest purity improvement
4. Divide the data into two groups based on that split
5. Recursively repeat for each group until a stopping condition is met (pure leaves, max depth, minimum samples, etc.)

This is called the CART algorithm (Classification and Regression Trees). The key question is: how do you measure purity?

Information gain and entropy

Overfitting: the core problem with trees

A decision tree grown without constraints will perfectly memorise the training data. Every leaf will contain exactly one training example. Training accuracy: 100%. Test accuracy: terrible.

This is because unconstrained trees learn the noise in the training data as if it were signal.

The fixes:

Pre-pruning (early stopping): Set max_depth, min_samples_split, or min_samples_leaf to stop growing before the tree overfits.

Post-pruning: Grow the full tree, then remove branches that don't improve performance on a validation set (cost-complexity pruning, available in sklearn as ccp_alpha).

In practice, pre-pruning with max_depth=3 to 5 is a good starting point.

Try it yourself — split data with a decision tree

Decision trees for regression

Trees aren't just classifiers — they work for regression too. Instead of predicting a class at each leaf, predict the mean target value of all training examples that land there. The impurity measure changes from entropy to mean squared error — each split is chosen to minimise the weighted MSE across child nodes.

Regression trees are expressive but still piecewise constant — the prediction is a step function that jumps at every split. For smooth continuous predictions, ensemble methods like Random Forest and Gradient Boosted Trees produce much better results.

What's next

A single decision tree is unstable and limited in depth. Random Forest takes the instability and turns it into a feature: train hundreds of trees on different random subsets of the data and features, then average their predictions. The variance cancels out. What remains is a robust, high-accuracy model.