Model Explainability: SHAP & LIME

Why black boxes aren't good enough

A gradient-boosted tree ensemble or a deep neural network can produce excellent predictions while giving no human-readable account of why it made any particular one. For a recommendation system, that opacity is often tolerable. For a loan denial, a fraud flag, a medical risk score, or an insurance premium, it usually isn't — the people affected, the regulators overseeing the domain, and the engineers debugging a wrong prediction all need some explanation of what drove the output.

Model explainability is the set of techniques that produce that explanation after the fact, without requiring the underlying model to be inherently simple. The two dominant general-purpose techniques are SHAP and LIME, and they take different mathematical routes to roughly the same goal: attributing a prediction to its input features.

LIME: approximate locally with something simple

LIME (Local Interpretable Model-agnostic Explanations) explains a single prediction by perturbing the input slightly many times, observing how the black-box model's output changes across those perturbations, and then fitting a simple, interpretable model — typically a sparse linear regression — to that local neighborhood of perturbed inputs and outputs. The coefficients of that local linear model become the feature attributions: this feature pushed the prediction up, that one pushed it down, in the vicinity of this specific input.

LIME's strength is that it's model-agnostic and conceptually simple, and it works on any input type you can meaningfully perturb (tabular rows, image patches, text tokens). Its weakness is that the explanation is only valid locally, around the specific instance being explained, and the quality of the explanation is sensitive to how the perturbation and the local linear fit are configured — two reasonable-seeming configurations can produce noticeably different explanations for the same prediction.

SHAP: attribute by game-theoretic fairness

SHAP (SHapley Additive exPlanations) is grounded in Shapley values from cooperative game theory: treat each feature as a 'player' contributing to the 'payout' (the prediction), and compute each feature's average marginal contribution across every possible ordering in which features could be added to the prediction. This gives SHAP values a set of desirable mathematical properties that LIME doesn't guarantee — the attributions sum exactly to the difference between the actual prediction and a baseline, and each feature's contribution is computed consistently regardless of feature ordering.

Computing exact Shapley values is exponentially expensive in the number of features, so in practice SHAP uses efficient approximations, including TreeSHAP (exact and fast for tree-based models by exploiting tree structure) and KernelSHAP (a model-agnostic approximation using weighted linear regression, conceptually similar to LIME but with Shapley-consistent weighting).

SHAP vs. LIME

When explainability becomes a legal requirement

In several regulated domains, explainability isn't optional polish — it's a compliance obligation. Credit decisioning in many jurisdictions requires adverse action notices that specify the actual reasons for a denial, which in practice means the model's decision must be attributable to specific, communicable factors. The EU's GDPR includes provisions interpreted by many practitioners as granting a right to meaningful information about automated decisions with significant effects on individuals. Insurance underwriting, healthcare risk scoring, and hiring algorithms face similar scrutiny in various jurisdictions, sometimes backed by specific audit or disclosure requirements.

In these settings, explainability techniques aren't just a debugging convenience for the ML team — they're part of the system's legal defensibility, which means the explanation method, its stability, and its documentation often need to withstand external audit, not just satisfy an internal reviewer.

What's next

Explainability tells you why a specific prediction happened; observability tells you whether the whole system producing predictions is healthy. Continue to ML System Observability to see how teams monitor pipelines, data quality, and latency beyond model accuracy alone.