A Churn Model Stakeholders Didn't Trust Until We Showed Our Work

We shipped a churn model with 0.81 AUC in the first sprint. The customer success team looked at the output, nodded politely, and kept doing exactly what they'd been doing before — working accounts by gut feel and whoever complained loudest. Three months later the model was still sitting there, technically correct and completely ignored.

Accuracy was never the blocker

I assumed the problem was adoption friction — a bad UI, a missing Slack integration, something mechanical. It wasn't. When I finally sat with two CS reps and watched them use it, the real objection came out: they didn't trust a score they couldn't argue with. If the model said an account was at 78% churn risk and a rep believed otherwise from a call last week, there was no way to reconcile the two, so they defaulted to ignoring the score entirely.

A model nobody can argue with is a model nobody will act on, no matter how accurate it is.

The fix wasn't a better model. It was per-prediction SHAP explanations surfaced right next to the score, in plain language: 'risk driven mainly by a 40% drop in weekly logins and two unresolved support tickets.' Now a rep could agree, disagree with a specific reason, or flag that the ticket was actually resolved and the data was stale. The score became something to argue with instead of something to obey or ignore.

What changed after that

• Weekly active usage of the model's dashboard went from near zero to daily across the CS team within two weeks.
• CS started feeding back disagreements, which surfaced a stale-ticket-status bug in the data pipeline we'd never have caught otherwise.
• Leadership stopped asking for accuracy numbers in reviews and started asking how many at-risk accounts were saved, which was the actual metric that mattered.
• We kept the explanations even after trust was established, because the feedback loop they created was more valuable than the churn predictions themselves.

The lesson wasn't about SHAP specifically. It was that a model's AUC answers a data scientist's question, and almost nobody downstream is asking that question. They're asking 'why should I believe this,' and if you don't answer it explicitly, the model quietly dies in a dashboard nobody opens.