LLM Summarization Quality Degraded Silently After a Model Update

Our pipeline summarized long support call transcripts into three-sentence briefs for account managers. It worked reliably for months. Then, after what we later confirmed was a routine model version bump on the provider's side, quality quietly dropped — and nothing alerted us, because nothing we monitored measured quality.

What we were actually monitoring

Latency, error rate, token usage, uptime. All green the entire time. The summaries were still being generated, still well-formed English, still three sentences. They just started missing the actual point of the call more often — flattening an escalation into a neutral summary, or focusing on a tangent instead of the resolution. None of our dashboards had a way to know that.

We monitored everything about the pipeline except the one thing it existed to do well.

How we actually caught it

An account manager flagged, almost as a complaint in passing, that the summaries 'felt off lately.' That anecdote is what triggered the investigation — not a dashboard, not an alert. We pulled a sample of fifty summaries from before and after the model update and had two humans blind-score them against the original transcripts. The before/after quality gap was obvious once we looked, and completely invisible before.

• We added a lightweight automated eval — a second LLM call scoring each summary against the transcript for factual coverage — that runs on a sample of outputs daily, not just at deploy time.
• We pinned the model version instead of auto-upgrading, and now review release notes before any upgrade instead of after a complaint.
• We added a human spot-check cadence that didn't exist before, because 'the pipeline is running' and 'the pipeline is working' turned out to be different claims.

The unsettling part is how long this could have run. We happened to catch it in two weeks because someone said something out loud. On a pipeline with less direct human contact downstream, this kind of regression could have run for months with every metric we tracked staying green the whole time.