Evaluation-Driven Development for LLM Apps
Why LLM apps need their own development discipline
Traditional software has deterministic behavior: given the same input, the same code produces the same output, and unit tests check exact expected values. LLM applications don't behave this way — outputs vary across calls even at temperature zero in subtle ways, 'correctness' is often graded rather than binary, and a prompt change that improves one test case can silently regress ten others. Evaluation-driven development (EDD) adapts the discipline of test-driven development to this reality: build a representative evaluation set early, run it on every meaningful change, and treat eval results — not vibes from a handful of manual spot-checks — as the primary signal for whether a change is actually an improvement.
Building the evaluation set
A useful eval set is built from a mix of sources: real production inputs (sampled and anonymized, since synthetic test cases alone systematically miss the long tail of how real users actually phrase things), deliberately adversarial or edge-case inputs targeting known failure modes, and regression cases captured specifically because a past version of the system failed on them — so a fix never silently regresses later. Eval sets should grow over time as new failure modes are discovered in production; a static eval set written once at the start of a project inevitably becomes a poor proxy for real-world behavior as the system and its usage evolve.
Grading: exact match, rubrics, and LLM-as-judge
Where outputs are structured or fall into a small discrete set (classification labels, extracted fields, tool call correctness), exact-match or programmatic checks work well and should be preferred — they're cheap, deterministic, and need no further validation. For open-ended generation (summaries, explanations, conversational responses), two main grading strategies exist: human rubric-based grading (reliable but slow and expensive at scale) and LLM-as-judge — using a separate, typically stronger LLM call to score outputs against a rubric. LLM-as-judge scales far better than human grading but introduces its own failure modes that need to be tracked, not assumed away.
def llm_judge_score(question, model_answer, reference_answer, judge_model):
rubric = """Score the model's answer from 1-5 against the reference answer:
5 = fully correct and complete
3 = partially correct, missing key details
1 = incorrect or irrelevant
Return only the integer score."""
prompt = f"{rubric}\n\nQuestion: {question}\nReference: {reference_answer}\nModel answer: {model_answer}"
score = int(judge_model.generate(prompt).strip())
return score
def run_eval_suite(eval_cases, app, judge_model):
scores = [llm_judge_score(c.question, app.run(c.question), c.reference, judge_model) for c in eval_cases]
return sum(scores) / len(scores), scores # mean score + per-case breakdown for regression trackingValidate the judge against human grading before trusting it
Grading strategies compared
| Grading method | Scalability | Cost | Reliability for open-ended tasks |
|---|---|---|---|
| Exact match / programmatic | Very high | Near zero | Only applicable to structured outputs |
| Human rubric grading | Low — bottlenecked by human time | High per example | High, if raters are well-calibrated |
| LLM-as-judge | High | Low-moderate (one extra LLM call) | Moderate — needs validation against human grades |
An aggregate eval score can mask severe per-segment failures
What's Next?
Evaluation-driven development is the feedback loop that makes every other technique in this space — prompting changes, fine-tuning, RAG tuning, agent guardrail calibration — actually measurable rather than guesswork, and it becomes increasingly essential as a system's behavior space grows too large for any individual to manually verify by inspection alone.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.