We Tested Quantized Model Quality Tradeoffs on Our Actual Workload

Published benchmarks for quantized models tend to report aggregate quality scores across broad tasks, which told us almost nothing useful for the client's actual workload: classifying support tickets into one of 40 fine-grained categories, a task with much tighter tolerance for subtle degradation than the general benchmarks those scores come from.

We ran the client's existing evaluation set — a few hundred hand-labeled tickets — against the full-precision model and three quantization levels, and measured category accuracy directly instead of trusting a generic benchmark that had nothing to do with this specific classification task.

What we found

Full precision:  94.2% category accuracy (baseline)
INT8 quant:       93.8% accuracy — negligible difference
INT4 quant:       89.1% accuracy — noticeable drop
INT3 quant:       76.4% accuracy — unacceptable for this task

INT8 was functionally free — within noise of full precision on this task, and it cut inference cost and latency meaningfully. INT4 degraded specifically on the categories that were already closest to each other semantically, meaning the quantization wasn't hurting accuracy evenly, it was collapsing exactly the distinctions this task cared about most.

A quantized model's aggregate benchmark score tells you almost nothing about how it'll behave on your specific decision boundary.

What we shipped

INT8 went to production, cutting infrastructure cost by roughly a third with accuracy indistinguishable from full precision on the actual task. INT4 looked tempting on the published general benchmarks and would have quietly degraded exactly the categories this client's business cared about getting right. The only way we found that out was running our own task-specific eval instead of trusting a number that was never measuring the thing we actually needed to know.