Feature Importance in Random Forests
Two different questions, two different importance measures
"How important is this feature?" sounds like one question but random forests answer it in at least two structurally different ways, and they can disagree substantially. Mean Decrease in Impurity (MDI), also called Gini importance, is computed directly from the trees during training. Permutation importance is computed after training by measuring how much performance degrades when a feature's values are shuffled. Knowing which one a tool reports, and why they diverge, is essential to not misreading feature importance plots.
Mean Decrease in Impurity (the default, and its trap
Every time a tree splits on a feature, that split reduces impurity (Gini impurity or entropy) by some amount. MDI importance for a feature sums up the impurity reduction from every split using that feature, across every tree, weighted by the number of samples reaching that split. It's fast — a free byproduct of training — but has a well-documented bias: it systematically favors high-cardinality features (continuous variables, or categoricals with many levels) because such features offer more possible split points to try, giving them more chances to find a spuriously good split, even if the feature is pure noise.
import numpy as np
from sklearn.ensemble import RandomForestClassifier
rng = np.random.RandomState(0)
n = 500
y = rng.randint(0, 2, n)
X = np.column_stack([
rng.randint(0, 2, n), # low-cardinality, genuinely informative
rng.normal(size=n), # high-cardinality, pure noise
])
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X, y)
print("MDI importances:", rf.feature_importances_)
# The noisy continuous feature often gets inflated MDI importance
# purely because it offers more candidate split thresholds,
# even though it has zero real relationship with y.
Permutation importance: a model-agnostic fix
Permutation importance sidesteps the cardinality bias by measuring the actual effect on predictive performance: take a trained model, shuffle one feature's column in the validation set (breaking its relationship with the target while preserving its marginal distribution), and measure how much a chosen metric (accuracy, AUC, RMSE) drops. A feature the model truly relies on causes a large performance drop when shuffled; an irrelevant feature causes little change, regardless of its cardinality. It's model-agnostic — it works for any fitted estimator, not just trees — at the cost of needing a held-out evaluation set and repeated re-scoring.
Correlated features distort both methods
| Property | MDI (Gini) importance | Permutation importance |
|---|---|---|
| Computed from | Training-time split statistics | Post-hoc performance drop |
| Needs held-out data | No | Yes (recommended) |
| Biased toward high-cardinality features | Yes | No |
| Model-agnostic | No — tree-specific | Yes — any model |
| Computational cost | Free (byproduct of training) | Requires repeated re-scoring |
What's next?
Both of these methods tell you how much a feature matters to the model's forest-wide behavior, but not how it got there — gradient boosted trees, the other dominant tree-ensemble family, build their predictive power through a fundamentally different sequential error-correction process worth understanding on its own terms.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.