Structured output from LLMs: JSON mode, schemas and what still breaks

Structured output means the model returns data in a fixed shape, usually JSON that matches a schema you supply, instead of free text you parse yourself. The provider’s decoder enforces the schema, so you get valid JSON with the keys and types you asked for. It does not check that the values are right, and a few cases still return output that does not match the schema.

JSON mode vs schema-constrained output

There are two levels. JSON mode asks for valid JSON and nothing more. Schema-constrained output (OpenAI and Claude call it structured outputs) enforces a JSON Schema. OpenAI’s guide says that only Structured Outputs ensure schema adherence and recommends always using them instead of JSON mode when possible. Older models, such as gpt-4-turbo and earlier, may use JSON mode instead.

Claude splits the feature in two. JSON outputs put the schema in output_config.format with type: "json_schema", and strict tool use (strict: true) validates tool names and inputs. OpenAI similarly lets you use a schema through function calling or through text.format: function calling when you connect the model to tools, text.format when you shape the reply to the user. For how tool calls fit with MCP, see MCP vs API.

Gemini accepts a response_format with a JSON mime type and your schema, and Pydantic (Python) or Zod (JavaScript) can generate that schema.

What each provider supports

Each provider takes a subset of JSON Schema, and a schema that works on one can be rejected on another. The table reflects the providers’ docs as of November 2026.

	OpenAI	Claude	Gemini
Required fields	All fields must be required; emulate optional with a null union	Supported	Supports required
additionalProperties	Must be false	Anything other than false is unsupported	Supported as a keyword
Unsupported or ignored keywords	allOf, not, dependentRequired, dependentSchemas, if, then, else; root cannot use anyOf	Recursive schemas, complex types within enums, external $ref, minimum, maximum, multipleOf, minLength, maxLength, array constraints beyond minItems of 0 or 1	Stripped, not rejected: minLength, maxLength, minProperties, maxProperties, uniqueItems, contains
Size limits	Up to 5,000 object properties, 10 levels of nesting	20 strict tools per request; “Schema is too complex” error	Large or deeply nested schemas can return 400 INVALID_ARGUMENT

The difference that bites is in how unsupported keywords are handled. Claude returns a 400 for an unsupported feature, while Gemini strips keywords such as minLength without an error, so a length rule you wrote is never enforced. Most of Claude’s SDK helpers can move some unsupported constraints into field descriptions and validate the response locally; the raw API does not.

What still breaks

Refusals. OpenAI’s response can include a refusal field, and since a refusal does not necessarily follow your schema, check for it before parsing. Claude returns stop_reason: "refusal" with a 200 status, bills the tokens and may not match the schema.

Truncation. When the response hits the token limit, the JSON stops mid-object. Claude’s docs say a max_tokens stop might be incomplete and not match your schema and advise retrying with a higher limit. OpenAI’s examples check for an incomplete status with the reason max_output_tokens.

First-request latency. OpenAI says the first request with a schema has additional latency while the schema is processed. Claude says the same while the grammar compiles, and caches compiled grammars for 24 hours from last use. Changing the schema’s structure invalidates the cache, but changing only a name or description does not.

Enum casing. Claude’s docs warn that it might return an enum value that differs only in capitalization, so compare case-insensitively.

Wrong values. The schema fixes the shape, not the content. OpenAI’s guide says outputs can still contain mistakes and suggests adjusting instructions, adding examples (see few-shot prompting) or splitting the task into simpler subtasks. A field filled with a plausible invented value is a hallucination that happens to parse.

Does a schema cost accuracy?

Tam and colleagues reported in August 2024 that they observed “a significant decline in LLMs reasoning abilities under format restrictions,” with stricter formats generally causing larger drops on reasoning tasks. It is one study of models from that period. If a task needs reasoning, test it with and without the constraint on your own data before assuming either result.

A validate-and-retry flow

Treat the provider’s constraint as the first layer, not the only one:

1. Request the schema with the provider’s structured output feature.
2. Check the stop reason or status for refusal and truncation before parsing.
3. Parse and validate in your own code, using the same schema (Pydantic’s model_validate_json() or Zod’s .parse() are the examples Google gives). Enforce the keywords the provider ignores.
4. Check meaning where you can: ranges, enum membership, references to ids that exist.
5. On failure, retry with a higher token limit or a tighter prompt, and log the failure.

Frameworks wrap this loop. See how LangChain, LangGraph and Pydantic AI handle typed output.

Testing structured output with HoneyHive

Keep the schema version and the raw response on each trace so failures are searchable; tracing captures LLM calls through OpenTelemetry. HoneyHive’s evaluator templates include a JSON Schema Validation evaluator, a Python evaluator that parses the output and checks it with the jsonschema library, returning true or false. A JSON Key Coverage template counts missing required fields. Python evaluators cover custom checks, and offline experiments run them over a dataset so you can compare a prompt or schema change before shipping. For the wider picture, see LLM evaluation.

FAQ

What is the difference between JSON mode and structured outputs?

JSON mode returns valid JSON. Structured outputs also enforce your schema. OpenAI's guide says "only Structured Outputs ensure schema adherence" and recommends them over JSON mode when possible.

Does structured output guarantee the values are correct?

No. It constrains the shape. Google's docs say "while output is syntactically correct JSON, always validate values in your application", and OpenAI's guide says outputs can still contain mistakes.

Can structured output fail to match the schema?

Yes, in two documented cases. A refusal need not follow your schema, and a response cut off at the token limit can be incomplete.

Does forcing JSON hurt reasoning?

One 2024 study, Let Me Speak Freely?, reported a significant decline in reasoning ability under format restrictions. It is a single study, so measure the effect on your own task.

How do I trace and test structured output in production?

Log each response with the schema version, then score it with a deterministic check. HoneyHive has a JSON Schema Validation evaluator template for this.