Data Labeling Quality Control Gaps That Quietly Hurt Model Performance

A classification model had plateaued at an accuracy ceiling for weeks, and the team's instinct was to keep attacking it as an architecture and feature engineering problem — bigger models, new features, different loss functions. None of it moved the needle meaningfully. The actual ceiling turned out to have nothing to do with the model. It was the labels.

What a manual label audit found

We finally pulled a random sample of a few hundred labeled training examples and had two internal people independently re-label them without seeing the original vendor labels. The disagreement rate between our re-labels and the original vendor labels was over 12%, concentrated heavily in a handful of categories with genuinely ambiguous boundaries that the labeling instructions hadn't addressed clearly enough.

A model can't be more accurate than its labels. If you haven't audited your labels, your accuracy ceiling might be a property of your labeling process, not your model.

Why nobody had checked sooner

The labeling had been outsourced early in the project, the labeling guidelines were written once and never revisited, and the categories where disagreement was highest had drifted slightly in their real-world meaning since the guidelines were first written. No single mislabeled example looked alarming in isolation — it's the kind of problem that's invisible until you deliberately sample and check, because nothing about a 12% noisy label rate produces an obvious error message anywhere in the pipeline.

• Random-sample audits of labeled data, done periodically, not just once at project kickoff.
• Inter-annotator agreement measured and tracked, not assumed — low agreement on a category is a signal to rewrite its guideline.
• Labeling guidelines treated as living documents that get revisited as edge cases accumulate, not a one-time spec.
• A model accuracy plateau should trigger a label audit as a first hypothesis, not a last resort after architecture changes fail.

We rewrote the guidelines for the worst-disagreement categories, relabeled the affected subset, and retrained on the cleaned data — and the accuracy ceiling we'd been fighting for weeks moved meaningfully in a single retrain. The architecture changes we'd tried before that hadn't been wrong exactly, they just couldn't fix a ceiling the data itself was imposing.