March 5, 2026 · 5 min read
Anomaly Detection False-Alarm Fatigue Almost Got the System Turned Off
A fraud anomaly system was technically working. It was also about to get disabled entirely, because the team monitoring it had stopped believing its alerts.
By the time I got pulled into this, the fraud ops team had a Slack channel with over 400 unread anomaly alerts and an informal rule nobody had written down but everyone followed: don't bother opening the channel unless someone escalates manually. The anomaly detection system was, by every modeling metric, working fine. It had also become background noise that nobody trusted enough to act on.
How it got there
The model flagged anything more than three standard deviations from a rolling baseline. That threshold had been reasonable at launch. Eighteen months later, with transaction volume up 3x and new payment types added without anyone revisiting the baseline, the system was firing 60-80 alerts a day. Investigating one took roughly ten minutes. Nobody had that kind of time, so the team quietly stopped trying, and real fraud started slipping through in the noise.
A detection system that cries wolf 80 times a day isn't a detection system anymore, it's a reason to ignore the one time it's right.
What actually fixed it
- Re-baselined the standard deviation threshold against current volume instead of the launch-era baseline, cutting alert volume by more than half overnight.
- Added a severity tier so genuinely extreme anomalies routed differently than borderline ones, instead of treating every flag identically.
- Built a feedback button directly in the alert so analysts could mark false positives in two seconds, which fed back into threshold tuning weekly instead of annually.
- Set an explicit review cadence for the thresholds themselves — quarterly, on the calendar — so this wouldn't silently drift again.
None of this involved a better model. The algorithm was fine the whole time. What broke was the operational discipline around it — nobody owned revisiting the threshold as the underlying data changed, so the system aged into uselessness in plain sight. The scariest part of the postmortem wasn't the false positives. It was realizing the team had been one bad quarter away from shutting the whole thing off and going back to pure manual review.