Structured Output
Making a model return JSON that always parses — solved at the format layer, and still wide open at the correctness layer.
When not to use it
- When a human reads the output. Prose is the right format for people. Don't schema your way into worse writing.
- When the task needs reasoning and you're constraining from the first token. Let it think in text, then structure in a second pass.
- When the shape is genuinely unknown. Forcing a schema onto something that doesn't have one produces confident nonsense in well-formed fields.
- As a substitute for validation. Schema-valid is not business-valid.
{"age": 900}parses fine.
Reach for something else instead
- Function calling — when you want an action rather than a value.
- Free text plus a parser — fine when the format is trivial and failures are cheap.
- Two-pass: reason then structure — more reliable on hard tasks, costs an extra call.
- Deterministic extraction — regex or a parser, when the input is actually regular. Faster and exact.
Sources & further reading
- Willard & Louf (2023), Efficient Guided Generation for Large Language Models — the finite-state machine approach behind most constrained decoding.
- Geng et al. (2023), Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning — grammar-constrained generation, and its costs.
- Tam et al. (2024), Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models — the evidence that constraint can degrade reasoning.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Believing valid means correct. It means parseable. Those are different bugs and only one of them wakes you up.
- Cryptic field names. The names are prompts;
xandcustomer_sentimentproduce different answers. - No
"unknown"or"other"option, which forces a fabrication and stores it as fact. - Constraining a reasoning-heavy task from the first token and losing quality to the grammar.
- Skipping business validation because the schema passed. Types are not semantics.
At a glance
FieldAI Agents
What's guaranteedthe format
What isn'tthe content
Best mechanismconstrained decoding
Schema designfield names are prompts
Hidden costconstraint can degrade reasoning
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Structured output vs. "reply in JSON please" — one is enforced at the sampler, the other is a request. The second works most of the time, which is the worst possible reliability.