Benchmark Scores Rarely Predict How a Model Performs on Your Actual Use Case

We ran a client's document-extraction task on the top three models by general benchmark score at the time. The one that scored lowest on the public leaderboards produced the highest accuracy on our actual test set by a meaningful margin — around 12 percentage points over the leaderboard leader.

This isn't a fluke, it's structural. Public benchmarks measure broad general capability across a mix of tasks nobody's client actually has. Your task is narrower and weirder than the benchmark distribution — maybe it's extracting line-item totals from inconsistently formatted invoices, which rewards a very specific kind of pattern-matching that doesn't correlate cleanly with, say, graduate-level reasoning questions.

A leaderboard tells you how a model performs on the leaderboard's tasks. It tells you almost nothing about how it performs on yours.

What actually predicts performance

• A held-out eval set built from real examples of your actual task, ideally fifty to a few hundred cases with known-correct answers.
• Testing on the messy, malformed real inputs, not clean ones — benchmark-topping models often degrade faster than expected on genuinely bad input.
• Measuring the specific failure mode that costs you the most, not aggregate accuracy — a model that's 90% accurate but fails silently is worse than one that's 85% accurate but fails loudly.

The eval harness isn't optional

Every client who skipped building a task-specific eval set and went with the benchmark leader ended up re-running the comparison themselves after a disappointing launch, at which point they'd already spent weeks building prompts and integration code around the wrong model. The twenty hours it takes to build a real eval set upfront is cheaper than that rework, every single time, and it's the first thing I now insist on before any model gets picked.