Evaluating Code-Generation LLMs
Why code generation needs its own evaluation methodology
Evaluating a code-generation model can't rely on the text-similarity metrics used elsewhere in NLP (BLEU, ROUGE) because code has a property most text doesn't: it's executable, and two functions that look completely different as strings can be functionally identical. This single fact reshapes the entire evaluation discipline — the field converged on functional correctness via execution (does the generated code pass a held-out test suite?) rather than surface-level similarity to a reference solution.
pass@k: the standard metric
The dominant metric, introduced with OpenAI's HumanEval benchmark, is pass@k: generate k samples per problem, and count the problem as solved if at least one sample passes all unit tests. Because naively sampling k times and checking is a biased, high-variance estimator, the actual formula uses an unbiased estimator that generates n ≥ k samples and computes the expected pass rate over all size-k subsets:
$$\text{pass@k} = \mathbb{E}_{\text{problems}}\left[1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\right]$$
where n is the total samples generated, c is the number that passed, and k is the subset size being evaluated. This lets researchers report pass@1 (single-shot accuracy) and pass@100 (best-of-100) from the same sampled pool without regenerating.
Benchmark contamination is a real, persistent problem
Beyond toy problems: SWE-bench and repo-scale evaluation
HumanEval-style benchmarks test isolated, self-contained functions — not the reality of professional software engineering, which mostly involves modifying existing, large codebases. SWE-bench addresses this directly: it sources real GitHub issues and their accepted pull-request fixes from popular open-source repos, gives the model the issue description and the repo state before the fix, and checks whether the model's generated patch passes the project's actual test suite. This is a dramatically harder and more realistic task, and even frontier models historically solved well under half of SWE-bench's problems without agentic scaffolding (retrieval, iterative tool use, test-running).
from math import comb
def pass_at_k(n: int, c: int, k: int) -> float:
"""Unbiased estimator of pass@k given n samples, c of which passed."""
if n - c < k:
return 1.0
return 1.0 - comb(n - c, k) / comb(n, k)
# Example: generated 20 samples, 6 passed all unit tests
print(pass_at_k(n=20, c=6, k=1)) # estimated pass@1
print(pass_at_k(n=20, c=6, k=10)) # estimated pass@10Evaluating beyond pure correctness
A model can produce correct-but-poor code: no error handling, O(n²) where O(n log n) was trivial, or style that violates the project's conventions. Production evaluation suites increasingly layer additional checks on top of pass@k — static analysis (linting, complexity metrics), security scanning for injected vulnerabilities, and LLM-as-judge rubric scoring for readability and idiomaticity — since a code review bot that only checks "does it run" misses most of what a human reviewer actually cares about.
Code-generation benchmarks compared
| Benchmark | Task shape | Realism |
|---|---|---|
| HumanEval | Isolated function from docstring | Low — toy, contamination-prone |
| MBPP | Short crowd-sourced programming problems | Low-medium |
| SWE-bench | Fix a real GitHub issue in a real repo | High — closest to actual engineering work |
| LiveCodeBench | Continuously refreshed competitive-programming problems | Medium-high, contamination-resistant |
What's Next?
As code-generation models get folded into agentic coding tools, evaluation is shifting further still — toward measuring full task completion (did the PR get merged, did the deployed service stay healthy) rather than isolated patch correctness.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.