Model Drift We Only Caught Because of Boring Monitoring

Nobody asks for a model monitoring dashboard with any enthusiasm. It's the project everyone agrees is a good idea and nobody wants to prioritize, because it doesn't ship a feature and it doesn't move a metric anyone's reporting on this quarter. We built one anyway on a lead-scoring model, mostly out of habit, and three months later it's the only reason we caught a drift that had been quietly eroding accuracy the entire time.

Nothing about the model or the code changed

That's what made this one genuinely unsettling to debug. No deploy touched the model. No feature pipeline changed. The drift came entirely from the world changing underneath the model — the client had shifted ad spend toward a new channel mid-quarter, which shifted the distribution of incoming lead characteristics in ways the training data from six months earlier had never seen. The model kept making confident predictions. It just got steadily worse at being right.

A model doesn't need a bug to fail. It just needs the world to keep moving while the training data stays still.

What the dashboard actually showed

The signal wasn't a dramatic accuracy cliff — it was a slow drift in the prediction distribution, scores clustering differently than the historical baseline, visible weeks before anyone downstream would have noticed anything felt off in the actual lead quality. We'd set up the dashboard to flag exactly that kind of distributional shift, almost as an afterthought, without a strong conviction it would ever catch anything real.

Week 1-4:  prediction distribution matches training baseline
Week 5-8:  gradual shift in score clustering, no alert threshold crossed
Week 9:    distribution shift flagged, investigation starts
Week 10:   root cause found — new channel's leads look structurally
           different from training data, model retrained on refreshed data

The retrain took an afternoon once we knew what to look for. The three months before that, the client had been making budget decisions on scores that were quietly becoming less reliable, and there was no code change, no incident, no obvious trigger that would have pointed anyone there without the monitoring already running. Boring infrastructure that nobody's excited about is usually the infrastructure that actually catches the failures nobody would have seen coming.