Our Data Quality Monitoring Caught Nothing for Months, Then Everything

We'd set up data quality monitoring on a client's event pipeline — row count checks, null rate alerts, schema drift detection, the works. It ran clean for four months. Then we discovered, by accident, that a secondary conversion event had been silently broken the entire time. The monitoring never fired a single alert.

What broke

A tag configuration change had switched a conversion event's trigger condition in a way that cut its firing rate by roughly 85%, but didn't stop it entirely. It still fired, just far less often — a slow leak, not a cliff. Every alert we'd built was tuned to catch a cliff: row counts dropping to zero, null rates spiking past some threshold, a table going stale. An 85% volume drop that happened gradually over two release cycles looked, to every threshold we'd set, like normal week-to-week variance.

We built monitoring to catch the pipeline breaking. We never built monitoring to catch the pipeline quietly getting worse.

How we actually found it

A marketer noticed a specific campaign's conversion count looked low compared to her own gut sense of how the campaign was performing, and asked us to check. That's it. No alert, no dashboard anomaly, just a human's intuition against the numbers not matching a different human's intuition about the campaign.

• Threshold-based alerts catch cliffs well and gradual decay almost never — they need a trend-based companion, not just a point-in-time check.
• We added week-over-week percentage change monitoring per event, not just absolute row counts, which would have caught this within a week.
• We added a monthly manual spot-check comparing a sample of events against what a human expects, specifically because automated monitoring has blind spots by design.
• Any tag or trigger configuration change now requires a before/after volume comparison in the PR, not just a QA click-test.

The monitoring wasn't broken. It was doing exactly what it was built to do. The lesson is that 'data quality monitoring' is a category, not a single implementation, and ours covered one failure mode while leaving the more common one — slow, partial degradation — completely uncovered.