Code Generation Models

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Code as a language model's native modality

Code generation models are language models trained to produce source code from natural language instructions, partial code, or both. The surprising discovery of the GPT-3/Codex era was that the same autoregressive transformer architecture used for natural language, trained on large corpora of public source code (GitHub repositories), learns to generate syntactically valid, often semantically correct code — because code, like natural language, is a sequence of tokens with strong statistical structure, just with much stricter syntactic rules and a much less forgiving notion of 'close enough.'

Training data and tokenization specifics

Code models are typically trained on a mix of permissively-licensed public code repositories, natural-language code comments/docstrings (providing paired code-description data), and increasingly, synthetic data generated by having a strong model produce and self-verify code solutions. Tokenization matters more for code than for prose — whitespace (especially Python's significant indentation) and special characters ({, (, _) carry syntactic meaning, so code-aware tokenizers either preserve whitespace tokens explicitly or use byte-level BPE that naturally handles arbitrary characters without an out-of-vocabulary problem. Models like Codex, CodeLlama, and StarCoder deliberately continue-pretrain a language-model backbone specifically on code-heavy data rather than training from scratch, leveraging general language understanding (docstrings, variable naming conventions, reasoning in comments) learned from natural language pretraining.

code_generation_inference.py

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("bigcode/starcoder2-7b")
model = AutoModelForCausalLM.from_pretrained(
    "bigcode/starcoder2-7b", torch_dtype=torch.bfloat16, device_map="auto"
)

prompt = '''def is_palindrome(s: str) -> bool:
    """Return True if s is a palindrome, ignoring case and non-alphanumeric characters."""
'''

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=80,
    temperature=0.2,     # low temperature -- code correctness favors determinism over creativity
    do_sample=True,
    stop_strings=["\ndef ", "\nclass "],   # stop at the next top-level definition
    tokenizer=tokenizer,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Fill-in-the-middle: why code models need bidirectional context

Natural language is usually generated left-to-right, but real coding happens in the middle of existing files — a developer's cursor sits inside a function with code both before and after it. Fill-in-the-Middle (FIM) training reformulates the standard autoregressive objective by splitting documents into prefix, middle, and suffix segments during training, and training the model to predict the middle segment given both the prefix and suffix as context (often rearranged as <prefix> <suffix> <middle> so the model still trains autoregressively, just over reordered segments). This single training trick is what makes modern code models usable as IDE autocomplete — without it, a purely left-to-right model can only extend code forward, never fill a gap with awareness of what comes after.

Training objective	What it enables	Example models
Standard left-to-right autoregressive	Generate code continuing from a prompt	Early Codex, GPT-based code models
Fill-in-the-Middle (FIM)	Insert code given both prefix and suffix context	CodeLlama, StarCoder2, most modern IDE autocomplete
Instruction-tuned / RLHF on code tasks	Follow natural-language coding instructions reliably	GPT-4, Claude, CodeLlama-Instruct

Pass@k reframes evaluation around real developer workflow

Code generation is typically evaluated with pass@k: the probability that at least one of k independently sampled completions passes a set of unit tests, reflecting that developers often look at a few suggestions and pick (or lightly edit) the best one rather than accepting a single greedy completion blindly. This is a meaningfully different evaluation philosophy than typical NLP accuracy metrics, and it's why code models are usually sampled with some temperature rather than greedy decoding during evaluation.

Syntactic validity doesn't imply correctness

A code model can generate syntactically perfect, idiomatic-looking code that is subtly wrong — an off-by-one error, a misunderstood edge case, a security vulnerability (SQL injection, unescaped input) — and these failures are far harder to catch by eye than a natural-language factual error, because the code 'looks right.' Production use of generated code requires test execution, static analysis, and security scanning as a mandatory pipeline stage, not optional review.

What's Next?

Generating a single function from a prompt is one end of a spectrum; the harder and more ambitious goal of synthesizing entire correct programs from high-level specifications pushes these same techniques considerably further.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →