Structured Outputs & JSON Mode

Why 'just ask for JSON' isn't reliable

The naive approach to getting structured data from an LLM is to ask for it in the prompt: "Respond only with valid JSON matching this format." This works most of the time, which is exactly the problem — most of the time isn't good enough for code that parses the response programmatically. Models under this approach occasionally wrap the JSON in markdown fences, add a trailing explanation sentence, use a slightly different key name than requested, or produce a trailing comma that breaks a strict parser.

For a human reading the output, these are trivial slip-ups. For a program calling JSON.parse() on the response, each one is a crash. At scale — thousands of calls a day — even a 1-2% malformed-output rate becomes a constant stream of parsing failures.

Two different mechanisms, often confused

JSON mode is a lighter-weight guarantee: the API constrains the output to be *syntactically valid JSON* — the braces balance, strings are quoted correctly, there's no stray prose — but it does not guarantee the JSON matches any particular schema. You still need to validate the shape of the resulting object yourself.

Structured outputs (schema-constrained generation) is a stronger guarantee: you supply a schema (typically JSON Schema), and the API constrains generation so the output is not just valid JSON, but valid JSON conforming to that exact schema — the right keys, the right types, enums restricted to allowed values, required fields always present.

Both work by intervening at the token-generation level, not by asking nicely in the prompt.

How constrained decoding actually works

Under the hood, structured output generation uses grammar-based (constrained) decoding. At each step of generating the response, the model produces a probability distribution over possible next tokens, as always. But before a token is chosen, the decoder masks out any token that would make the output violate the schema — if the schema says the next field must be a number, tokens that would start a letter or a quote are given zero probability, regardless of what the model's raw distribution preferred.

This is fundamentally different from prompting. Prompting tries to *persuade* the model to produce the right format by describing it in words — the model can still ignore or misread the instruction. Constrained decoding makes it *structurally impossible* for the model to emit a token that breaks the schema; the only freedom left to the model is choosing among the tokens that remain valid.

What structured outputs don't fix

Constrained decoding guarantees the *shape* of the output is valid. It says nothing about whether the *content* is correct. A model forced to produce a schema-conforming JSON object can still:

- Put a plausible-but-wrong number in a required numeric field.
- Select a valid enum value that doesn't actually match the input (e.g., picking 'USD' when the invoice was in EUR, because it misread the document).
- Hallucinate a required field with a made-up value when the source material didn't actually specify one, since the schema requires the field to be present.

Schema validation and content correctness are separate concerns. You still need business-logic validation, range checks, and — for anything consequential — human or automated review of the actual values, not just confirmation that the JSON parses.

Prompting for JSON vs. JSON mode vs. schema-constrained structured outputs

What's next

Structured outputs are one piece of making an agent's individual calls reliable. The data those calls operate on often has to come from somewhere — the next lesson is a deep dive comparing vector databases (Pinecone, Weaviate, Qdrant, pgvector), the storage layer behind retrieval-augmented agents and RAG systems.