A model can achieve excellent accuracy while remaining a black box about *why* it makes the predictions it does. That gap matters for more than curiosity: regulators may require an explanation for a loan denial, a doctor needs to know why a model flagged a scan as high-risk before trusting it, and an engineer debugging a model that quietly learned to rely on a leaked, spurious signal needs some way to catch that before deployment, not after.

Feature importance methods try to answer one specific, narrower question than full interpretability: of all the input features, which ones most influenced this model's predictions, and by how much? That's a useful and tractable question even when fully explaining a model's internal reasoning is not.

Built-in importance from tree-based models

Tree-based models (decision trees, random forests, gradient boosted trees) have a natural, built-in notion of feature importance, because the training process itself repeatedly chooses which feature to split on at each node, based on how much that split reduces prediction error (or impurity). A model's built-in feature importance is typically computed by summing up, across every split in every tree, how much that feature's splits reduced error, weighted by how many samples passed through that split.

This is essentially free to compute — it falls directly out of how the model was already built — which is why it's usually the first importance method people reach for with tree-based models. But it comes with a well-known bias: features with many possible split points (high-cardinality numeric or categorical features) tend to get inflated importance scores simply because they offer the tree more opportunities to find a useful-looking split, not necessarily because they're genuinely more predictive.

python

Permutation importance: a model-agnostic alternative

Permutation importance works completely differently, and crucially, works for *any* trained model, not just tree-based ones. The idea: take a trained model and a validation set, measure its performance normally, then shuffle (permute) the values of a single feature column — destroying any real relationship between that feature and the target while keeping everything else intact — and measure performance again. If shuffling a feature barely changes performance, the model wasn't relying on it much. If performance drops sharply, the model was leaning on that feature heavily.

Because this only requires being able to make predictions and measure performance, it works identically for a random forest, a neural network, or a linear model, and it directly measures impact on the metric you actually care about rather than an internal training statistic.

python

Built-in vs. permutation feature importance

Property	Built-in tree importance	Permutation importance
Works on any model type	No -- tree-based models only	Yes -- any trained model
Computation cost	Essentially free, already computed during training	Requires re-running predictions many times
Bias toward high-cardinality features	Yes, a known issue	Largely avoided
Measures impact on	Internal training statistic (impurity reduction)	Actual chosen evaluation metric directly

Note

A feature ranking as highly important only tells you the model relies on it for prediction — it says nothing about whether that feature *causes* the outcome. A classic failure mode: a model predicting hospital readmission ranks "number of past hospital visits" as hugely important, which is true and useful for prediction, but doesn't mean reducing someone's past visit count would reduce their readmission risk — the feature is a correlate and consequence of underlying health status, not a lever you could pull. Feature importance answers "what did the model use," never "what would changing this variable actually do in the real world" — that second question requires causal inference methods, which are a genuinely different field with different assumptions and guarantees.

Where this fits in a real workflow

Feature importance is most valuable as a sanity check and debugging tool, not a final answer. If a model ranks a feature that shouldn't plausibly matter (say, a row ID or a timestamp of when data was collected) as highly important, that's often a sign of data leakage — a variable that accidentally encodes information about the label that wouldn't be available at real prediction time. Catching that before deployment, via feature importance, is far cheaper than catching it after the model is already live and quietly underperforming in production.

What's next

Interpretability tools help you trust a single trained model, but real ML work involves many training runs with different hyperparameters, data versions, and architectures. Continue to ML Experiment Tracking to see how teams keep that process reproducible instead of relying on memory and scattered notebook cells.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →