Evals Should Be a Product Requirement, Not an Engineering Afterthought

A product spec landed on my desk once that said, in full, roughly: 'Use AI to summarize the call transcript for the sales rep.' No definition of what made a summary good. No example of a summary a stakeholder would reject. No acceptable length, tone, or content priority. Engineering built something, shipped it, and the first round of stakeholder feedback was a list of complaints that read like a spec that should have existed before a single line of code was written.

Eval criteria are a product decision, not a technical one

What makes a summary 'good' is a judgment about what the sales rep actually needs from it — should it prioritize next steps over sentiment, should it flag objections even if brief, should it be three sentences or a bulleted list. Those are product questions. Handing them to engineering as 'figure out what good looks like' means engineering is making product decisions by default, usually without realizing it, usually inconsistently across different parts of the feature.

If the product spec doesn't define what a good output looks like, engineering will define it for you, by accident, one implementation detail at a time.

What a spec with real eval criteria looks like

• A set of ten to twenty real example inputs with what a stakeholder-approved output looks like for each.
• Explicit priorities stated as rules, not vibes — 'always mention pricing objections even if the call didn't dwell on them' is a testable rule, 'make it useful' is not.
• Defined failure modes to avoid, written down before launch, not discovered from complaints after.
• A review checkpoint where a product owner scores a sample batch of real outputs before the feature is considered done, not just 'demoed.'

This is cheaper earlier than it is later

Once we rebuilt that spec with ten real example transcripts and stakeholder-approved summaries attached, the rebuild took less time than the original build, because engineering had an actual target to hit instead of a vibe to interpret. Writing the eval criteria upfront isn't extra process on top of the product spec — in any feature that uses an LLM, it is the product spec. Everything else is implementation detail.