A Computer Vision Model's Edge Case That Only Showed Up in Production

The product image classifier for a retail client looked great on paper: 96% test accuracy, clean validation curves, nothing alarming in the review. Two weeks after launch, a category manager flagged that the model was consistently misclassifying products photographed against a specific type of patterned background — a background that, it turned out, was overwhelmingly used in exactly one product category.

The test set hid a coverage gap, not an accuracy problem

That product category made up less than 2% of the training and test images. Overall accuracy stayed high because the model was fine on everything else, and the aggregate metric simply diluted a near-total failure on a small slice into statistical noise. Nobody reviewing the 96% number had any reason to suspect it was hiding a category where real-world accuracy was closer to 40%.

A 96% accurate model can still be completely wrong on a category that's underrepresented enough to vanish inside the aggregate.

• Always check per-category accuracy, not just overall accuracy, before shipping a classifier with meaningfully imbalanced categories.
• A small test set slice (under 5% of samples) can't give you statistical confidence on that category's real performance, even if it looks fine.
• Oversample underrepresented categories specifically for evaluation, even if training stays naturally weighted.
• Production monitoring by category, not just in aggregate, catches this class of failure fast instead of waiting for a human to notice.

The fix was targeted data augmentation for the underrepresented category and a monitoring dashboard broken out by category instead of one blended number. The real lesson was about review process: an aggregate accuracy metric in a model card is practically designed to hide exactly this kind of gap unless someone explicitly asks to see it sliced.