The Real Cost of False Positives in a Fraud Detection Model

A fraud detection model we built for an e-commerce client hit a recall number the team was genuinely proud of — catching the vast majority of actual fraudulent transactions. Three weeks after launch, the finance team flagged a problem the model's dashboard had never surfaced: a noticeable dip in legitimate revenue, entirely attributable to the model blocking real, paying customers it had wrongly flagged as fraud.

Why recall was the wrong north star metric

We'd optimized the threshold to maximize fraud recall, which is the natural instinct when the loss function in front of you is fraud dollars prevented. What that optimization didn't account for was the actual, measurable cost of a false positive — a legitimate customer wrongly blocked doesn't just fail one transaction, they often abandon the purchase entirely and in a meaningful share of cases don't come back. That cost was real revenue, happening every day, and it simply wasn't in the metric we'd been optimizing against.

A fraud model's recall tells you how much fraud it caught. It tells you nothing about how much legitimate business it also broke to get there.

What changed once we priced it correctly

We built an actual cost model — average fraud loss prevented per true positive, against average legitimate revenue and customer lifetime value lost per false positive — and used that to pick a threshold that minimized total expected cost rather than maximizing recall in isolation. The resulting threshold let through somewhat more fraud in absolute terms, and it was a clear net financial win because the false positive cost had been so much higher than anyone had actually quantified before.

Old threshold: recall 94%, false positive rate 3.1%
New threshold: recall 88%, false positive rate 0.9%

Net financial impact: positive, despite lower recall —
false positive cost per incident was ~6x the average fraud loss prevented

• Model threshold decisions need a real cost model behind them, not a single metric optimized in isolation.
• False positive cost on a fraud model isn't just the immediate blocked transaction — factor in customer churn and lifetime value lost.
• Recall and precision are both inputs to a business decision, not the decision itself.
• Finance and the model team need a shared cost framework before launch, not after a revenue dip triggers an investigation.

The model's accuracy metrics didn't change in any way that would show up in a typical ML evaluation report. What changed was asking finance for the actual dollar cost of a false positive before picking a threshold, instead of after a revenue dip forced the conversation. That one number should have been part of the original project scope, not a retrofit three weeks into production.