We Fine-Tuned When We Should Have Just Improved the Prompt

The task was tagging support tickets into one of about twenty categories, and the base model's accuracy wasn't great out of the box — around 71%. Someone on the team had read a compelling case study about fine-tuning and pitched it as the fix. We committed five weeks and a real training budget before anyone seriously tried improving the prompt first.

What the fine-tune actually bought us

It worked. Accuracy went to 89%. The problem is what we discovered afterward almost by accident: a better system prompt with five well-chosen few-shot examples per category, plus a structured output schema that forced the model to pick from an enumerated list instead of free-text, got us to 86% in an afternoon, with zero training cost and zero retraining burden every time the category list changed.

A three-point accuracy gain that costs five weeks and locks you into a static category list is not obviously a win over an afternoon fix that doesn't.

The real cost wasn't the training run

It was everything downstream of having a fine-tuned model: a retraining pipeline we now had to maintain, a model artifact we had to version and evaluate separately from the base model's updates, and a category taxonomy that was now baked into weights instead of living in an easily editable prompt. Three months later, support added two new ticket categories. The prompt-based approach would have handled that with a one-line edit. The fine-tuned model needed a retraining cycle.

• Fine-tuning made sense for tasks with a genuinely stable, large-scale pattern that prompting structurally can't capture — this wasn't that.
• It makes much less sense for anything whose taxonomy, format, or rules change more than once a quarter.
• Always run the cheap option — better prompt, few-shot examples, structured output — to its actual ceiling before committing to the expensive one.
• The ceiling of prompting is higher than most teams assume, especially once structured outputs are in the mix.

We kept the fine-tuned model for a few more months out of sunk-cost inertia, then quietly swapped back to the prompt-based version when the category list changed again. Nobody noticed the swap in production quality. Everybody noticed how much faster iteration got once the rules lived in an editable prompt instead of frozen weights.