A Multilingual Gap in an NLP Pipeline That Cost Real Accuracy

The sentiment classifier reported an overall accuracy north of 90% in every evaluation we ran. What that number hid, because it was an aggregate across the entire ticket volume, was that performance on Spanish-language tickets — roughly 18% of the client's support volume — was closer to 65%. Nobody caught it for two reporting cycles because the dashboard only ever showed the blended number.

How the gap got built in

The classifier had been fine-tuned on a labeled dataset that was, it turned out, almost entirely English. Nobody had done that deliberately — the labeling vendor had defaulted to English tickets because that's what their labeling team was staffed for, and nobody on our side had specified otherwise or checked the language distribution of the training set after the fact.

A blended accuracy number is a great way to hide a subgroup that's failing completely.

Once we segmented eval metrics by detected language, the gap was undeniable and genuinely alarming — the model was essentially guessing on a fifth of all incoming tickets, and those tickets were disproportionately being misrouted to the wrong urgency queue as a result.

• Always segment model evaluation metrics by relevant subgroup, not just language — region, channel, and customer tier all hide the same kind of gap.
• Check the language distribution of a labeled training set against the real production distribution before trusting a fine-tune.
• A labeling vendor's default assumptions become your model's default blind spots unless you specify otherwise explicitly.
• Blended metrics are a reporting convenience, never a QA standard.

The fix was straightforward once diagnosed — a proper bilingual fine-tuning set and a per-language eval gate before any future deploy. The harder fix was cultural: we added a standing rule that no model ships without a subgroup breakdown in the eval report, specifically because this exact failure mode is invisible until someone goes looking for it.