SHAP Values Explained
The question SHAP answers
For a single prediction, SHAP (SHapley Additive exPlanations, Lundberg & Lee 2017) answers: how much did each feature contribute, in this specific instance, to pushing the prediction away from the model's average/baseline output? Unlike global feature importance (which answers 'what matters across the whole dataset'), SHAP gives a per-prediction, per-feature attribution that sums exactly to the difference between this prediction and the baseline — a property called local accuracy that most other explanation methods don't guarantee.
Grounded in cooperative game theory
SHAP values are an application of the Shapley value, a 1953 solution concept from cooperative game theory for fairly dividing a payout among players based on their contribution. Here, 'players' are features and the 'payout' is the prediction. The Shapley value for feature $i$ averages its marginal contribution to the prediction across every possible subset/ordering of the other features: $\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!(|F|-|S|-1)!}{|F|!} [f(S \cup \{i\}) - f(S)]$. This exhaustive averaging is what guarantees SHAP's fairness properties (symmetry, dummy feature gets zero, additivity) — but computing it exactly is exponential in the number of features.
Why TreeSHAP made this practical
Exact Shapley values are intractable for more than ~20 features by brute force. TreeSHAP exploits the specific structure of decision trees to compute exact SHAP values in low-order polynomial time (roughly $O(TLD^2)$ for $T$ trees, $L$ leaves, depth $D$) rather than exponential time, by tracking which feature subsets each path in the tree is 'consistent with' as it traverses the tree structure. This is the reason SHAP became standard specifically for gradient boosted trees and random forests — it's the one model family where exact, fast Shapley computation is actually solved.
import shap
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
model = xgb.XGBClassifier(n_estimators=100, max_depth=4).fit(X, y)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)
# Local accuracy: base value + sum of SHAP values == model's raw output
i = 0
print("Base value:", explainer.expected_value)
print("Sum of SHAP values for row 0:", shap_values[i].sum())
print("Model raw margin output for row 0:", model.predict(X.iloc[[i]], output_margin=True)[0])
# Global importance is just the mean absolute SHAP value per feature
import numpy as np
mean_abs_shap = np.abs(shap_values).mean(axis=0)
top_features = X.columns[np.argsort(mean_abs_shap)[::-1][:5]]
print("Top 5 features by mean |SHAP|:", list(top_features))
SHAP explains the model, not necessarily the world
| Method | Scope | Theoretical guarantee | Typical cost |
|---|---|---|---|
| Permutation importance | Global | None specific | Moderate |
| MDI (Gini) importance | Global | None, biased by cardinality | Free |
| SHAP (TreeSHAP) | Local (and aggregable to global) | Shapley fairness axioms | Polynomial, tree-specific |
What's next?
SHAP tells you how much each feature mattered for one prediction, but it doesn't directly show the *shape* of a feature's relationship with the outcome across its whole range. Visualizing that marginal relationship — holding other features fixed and sweeping one feature's value — is the complementary question that partial dependence plots are built to answer.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.