Structured output solves syntax, not truth. A model can return JSON that matches your schema and still select the wrong customer, misuse an enum, or request an unauthorized refund. Treat generation as the first step in a typed boundary: enforce the provider schema, validate business semantics, check policy, and only then allow downstream effects.
Retry only failures that a new generation can plausibly repair. Route refusals, exhausted attempts, and policy violations into explicit terminal states instead of hiding them behind a generic parser error.
Use four validation gates
| Gate | Question | Example failure | Correct response |
|---|---|---|---|
| Transport | Did the request complete with the expected response type? | Timeout or truncated stream | Retry under request policy or return unavailable |
| Schema | Does the value match required types, enums, and shape? | amount is a string | One bounded repair attempt or invalid output |
| Semantics | Is the value valid for current application state? | Order is already refunded | Reject; refresh state only when safe |
| Policy | May this principal perform this action? | Refund exceeds approval limit | Deny or require approval outside the model |
OpenAI supports strict JSON Schema output for compatible models while noting that only a subset of JSON Schema is supported. Anthropic tool definitions use an input_schema JSON Schema object. Provider enforcement reduces malformed output, but the application still owns semantic and authorization checks. See the OpenAI response-format reference and Anthropic tool-use implementation guide.
Design the schema as an application contract
Prefer a small tagged result over an object full of optional fields:
{"type": "object","additionalProperties": false,"properties": {"decision": { "enum": ["approve", "reject", "needs_review"] },"reason_code": { "enum": ["policy_match", "missing_evidence", "limit_exceeded"] },"evidence_ids": { "type": "array", "items": { "type": "string" } }},"required": ["decision", "reason_code", "evidence_ids"]}
Do not ask the model for values the server already knows, such as user identity, tenant, current time, or permission level. Join model output with trusted server state after validation.
Version the schema. Store schema, model, prompt, and validator versions on the task trace. A compatible provider API does not guarantee identical model behavior across snapshots; OpenAI recommends pinned model versions and evals when consistency matters. See its backward-compatibility guidance.
Make failure states explicit
type StructuredResult<T> =| { state: 'accepted'; value: T; attempt: number }| { state: 'refused'; reason: string }| { state: 'invalid'; errors: string[]; attempts: number }| { state: 'unavailable'; retryAfterMs?: number };
Keep refusal separate from schema failure. A refusal may be a valid response to the task, not broken JSON. Keep transport failure separate from invalid content so the caller applies the right retry policy.
Retry with a bounded repair envelope
One repair attempt is often enough to distinguish a transient formatting failure from a broken contract. Send validation errors, the original schema version, and the original input; do not silently broaden the schema.
Do not retry a policy denial, missing source data, an invalid user-supplied resource, a side effect with uncertain outcome, or repeated semantic failure with unchanged state. If a model call precedes a consequential tool, the idempotency and retry boundary must own uncertain execution.
Test the contract, not just happy JSON
valid object → acceptedextra property → invalidunknown enum → invalidvalid order ID from another tenant → policy deniedalready-refunded order → semantic invalidprovider refusal → refusedtimeout before response → unavailabletimeout after side effect → uncertain, never regenerate action
Track acceptance, refusal, schema failure, semantic failure, policy denial, repair success, and latency per attempt. Put them on one task trace using the production LLM observability schema.
The release gate is not “99% valid JSON.” It is task success after every gate, with dangerous false accepts reported separately. Preserve failures as regression cases with the LLM evaluation dataset workflow.
Use structured outputs when downstream code needs a stable contract. Use ordinary text when the user is the consumer and rigid structure does not improve the task.
