A Small Fine-Tuned Model Beat GPT-Class Models on Our Specific Task

The task was classifying incoming leads into one of fourteen categories based on a short free-text description, at a volume high enough that cost per call actually mattered. We started with a frontier model behind a well-tuned prompt, got to roughly 82% accuracy, and assumed the ceiling for improvement was a bigger or newer frontier model. It wasn't.

Why the narrow task mattered

Frontier models are generalists, trading some depth on any single narrow task for breadth across everything. Our task was the opposite of general — fourteen fixed categories, a consistent input format, and several thousand historical labeled examples sitting in the CRM that nobody had thought to use as training data. That's close to the ideal setup for a small fine-tuned model: narrow, stable, with real labeled data available.

A frontier model is a generalist competing on a task that rewards specialization. That's a fight the generalist doesn't need to lose, but often does.

The actual comparison

• Frontier model with tuned prompt: 82% accuracy, roughly $0.004 per classification at our volume.
• Small fine-tuned model trained on 4,000 historical labeled leads: 91% accuracy, roughly $0.0002 per classification.
• Latency dropped from around 900ms to under 100ms, which mattered for a real-time routing use case downstream.
• The fine-tuned model needed retraining when a category was added — a real cost the frontier approach didn't have, but one retraining run a quarter was a trivial price for the accuracy and cost gain.

This isn't an argument against frontier models

It's an argument against defaulting to them for every task without checking whether the task is actually narrow enough to reward specialization. Most of what we build genuinely needs a general reasoner. This task didn't — it needed a classifier, and we'd been using a very expensive, very capable reasoner to do a classifier's job. Once we recognized that, the fix was almost obvious, and the hardest part was admitting the frontier model was the wrong tool, not a better one.