Build the Eval Harness Before You Touch Prompt Tuning

A team I worked with was building a classification feature on top of an LLM and spent three weeks iterating on the prompt before anyone wrote down a single test case. Every change was evaluated by someone staring at five example outputs and going 'yeah, that looks better.' That's not iteration. That's astrology with extra tokens.

Vibes don't compose

The problem with eyeballing outputs is that it doesn't catch regressions. You fix the thing that was annoying you on Tuesday, and the fix quietly breaks three cases you weren't looking at, because nobody was looking at them — there was no fixed set of cases to look at in the first place. Three weeks of 'this feels better' produced a prompt that was, when we finally ran it against a real eval set, worse on 12% of cases than the one it replaced.

If you can't tell me the pass rate on fifty labeled examples, you don't know if your prompt change helped. You just know it changed.

What an eval harness actually needs to be

It doesn't need to be sophisticated. It needs to exist. A spreadsheet of fifty real inputs with the correct expected output, a script that runs the current prompt against all fifty and scores exact or fuzzy match, and a number you track over time. That's it. That number is the only thing that should be allowed to tell you a change is an improvement.

eval_set.jsonl  -> 50+ real examples with expected_output
run_eval.py     -> runs prompt against all examples, scores each
score: 0.0-1.0  -> the only number allowed to say 'this is better'

No eval run -> no merge. Full stop.

Once that team built the harness, two things happened fast. First, they found the actual regression and reverted it in a day instead of shipping it to production. Second, prompt iteration got faster, not slower, because every change was now a five-minute eval run instead of a group Slack thread of subjective impressions. The harness isn't overhead on top of prompt engineering — it's what makes prompt engineering an engineering discipline instead of a guessing game with better vocabulary.