Permutation Feature Importance
A model-agnostic definition of importance
Permutation feature importance defines a feature's importance as the drop in model performance when that feature's values are randomly shuffled, breaking its relationship with the target while leaving its marginal distribution untouched. Formally: $I_j = s - s_j^{perm}$, where $s$ is the baseline score (accuracy, AUC, $R^2$, etc.) on real data and $s_j^{perm}$ is the score after permuting feature $j$. Because it only requires the ability to predict and score — not any internal model structure — it works identically for linear models, trees, neural networks, or an ensemble of all three.
Train-set vs. held-out permutation importance
Permutation importance can be computed on the training set or a held-out validation/test set, and the two answer different questions. Training-set permutation importance reflects what the model *learned to rely on*, including spurious patterns it overfit to. Held-out permutation importance reflects what actually generalizes — a feature the model overfit to on noise will show high training-set importance but near-zero (or even negative) held-out importance, which is itself a useful overfitting diagnostic.
from sklearn.inspection import permutation_importance
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_california_housing
import numpy as np
data = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=0
)
model = RandomForestRegressor(n_estimators=200, random_state=0).fit(X_train, y_train)
result = permutation_importance(
model, X_test, y_test, n_repeats=20, random_state=0, scoring="r2"
)
order = np.argsort(result.importances_mean)[::-1]
for i in order:
print(f"{data.feature_names[i]:12s} "
f"importance={result.importances_mean[i]:.4f} "
f"+/- {result.importances_std[i]:.4f}")
# n_repeats=20 shuffles each feature 20 times to get a mean and std,
# since a single random shuffle is a noisy estimate.
Repeat the shuffle — a single permutation is a noisy estimate
n_repeats) and report both the mean importance and its standard deviation — treating a feature as genuinely important only when its mean drop is comfortably larger than its own run-to-run variance, not just larger than zero.The correlated-feature blind spot, revisited
Like MDI importance, permutation importance underestimates the importance of features that have close correlated substitutes: shuffling feature A barely hurts performance if feature B (highly correlated with A) is still intact and the model can lean on it instead. This is not a bug specific to permutation importance — it's a structural consequence of measuring importance via substitutability — but it's worth explicitly checking a correlation matrix before trusting a ranking that shows two theoretically related features both scoring surprisingly low.
| Importance method | Needs re-training? | Model-agnostic | Captures generalization (if on held-out set) |
|---|---|---|---|
| MDI (Gini) | No (training byproduct) | No — tree-specific | No |
| Permutation (train set) | No, just re-scoring | Yes | No |
| Permutation (held-out set) | No, just re-scoring | Yes | Yes |
What's next?
Importance and dependence methods describe how the model behaves in general, but a stakeholder asking 'what would need to change for this specific decision to flip?' needs a different kind of explanation entirely — one that proposes a concrete, minimal alternative input rather than describing a trend.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.