Validate a model's structured response at three levels: its shape, the relationships between its fields, and its support in the supplied evidence. A response can satisfy a schema while naming the wrong person, reversing a date range, or citing a source that does not contain the claimed fact. Schema adherence makes an output easier to consume; it does not make the information true.
This matters most when software acts on the result. A plausible owner name in a valid JSON object can be more dangerous than an obvious parsing error because downstream code may accept it without further inspection.
Give absence and conflict a legal representation
Consider extracting a maintenance window from two synthetic source lines:
s1: Window starts 2026-11-03 and ends 2026-11-04.
s2: The owner is unspecified.A useful candidate preserves the missing owner:
{
"start": "2026-11-03",
"end": "2026-11-04",
"owner": null,
"status": "supported",
"evidence": ["s1", "s2"]
}If the schema requires a nonempty owner string for every input, the system cannot honestly represent this document. Decide whether a missing field should be null, a separate status, or a validation failure. Distinguish missing information from contradictory information; two different owners in equally applicable sources require a conflict decision, not an arbitrary selection.
The OpenAI Structured Outputs guide distinguishes JSON mode from schema-constrained output, supports only a subset of JSON Schema, and explicitly says structured responses can still contain mistakes. It also describes refusal and incomplete-response paths. Those paths must be handled before treating the response as the application's normal result.
Check shape against the contract you actually support
Specify required keys, value types, allowed states, and treatment of additional properties. The JSON Schema object reference explains that listing a property does not make it required and that additional properties are permitted by default. Provider-specific strict modes can impose tighter rules than general JSON Schema.
Keep the schema and application types aligned. If an SDK derives one from the other, verify the emitted schema against the provider's supported features. If they are maintained separately, a schema change and a decoder change need a compatibility check.
Do not convert a refusal, a truncated response, or a parse failure into an empty object that looks like a successful extraction. Return an explicit processing state so the next layer can distinguish “no owner was present” from “the request never produced a usable extraction.”
Validate relationships that individual fields cannot establish
A syntactically valid date does not establish a valid interval. Once dates are parsed under the application's date and time rules, check that the start is no later than the end. If the source specifies a timezone, preserve it; if it does not, avoid inventing an offset merely to satisfy a timestamp type.
Other relationships include a total matching its line items, an identifier belonging to the stated account, and status: supported agreeing with the presence of usable evidence. Put deterministic rules in application validation when practical. A prompt can explain the rule, but the output consumer should not depend solely on the model remembering it.
Contract tests versus unit tests makes the same distinction at a service boundary: a correct message shape can contain a wrong business result. Structured generation moves that boundary; it does not remove it.
Compare the value with its evidence
A source ID must refer to an input that was actually supplied. A quoted span must exist in that source revision. Even both checks together do not prove that the source entails the extracted value: the quote may be about a different project or contain a negation.
For straightforward fields, a deterministic parser can verify a literal value. For ambiguous prose, review the proposed field beside the supporting passage and its scope. A second model may help prioritize review, but its agreement is another judgment, not independent ground truth.
The local example uses intentionally simple ISO date strings and exact source lines. It is narrow enough to check with ordinary Python conditions. General document extraction would need better parsing and a stronger evidence model than string membership.
What the four-candidate fixture found
We created four candidate objects by hand and ran shape, relationship, and fixture-specific support checks:
| Candidate | Shape check | Date relationship | Evidence check |
|---|---|---|---|
| Original supported extraction | Passed | Passed | Passed |
| Start and end reversed | Passed | Failed | Passed literal-value check |
| Invented owner “Mira” | Passed | Passed | Failed |
| Unknown source ID | Passed | Passed | Failed |
All four fit the narrow Python shape check. The reversed dates show why finding both date strings in the source is weaker than understanding their roles. The invented owner shows why an allowed string value can remain unsupported. The unknown ID shows why a plausible citation format is insufficient.
The assertions passed locally. No model generated these candidates, no API call ran, and no JSON Schema library was used. The fixture calibrates a small validation pipeline; it does not measure extraction accuracy or certify provider schema conformance.
Repair the failed layer
If shape validation fails, inspect the schema, decoder, response completion state, and provider compatibility. If relationships fail, return precise field-level feedback or route to review. If evidence is missing, retrieve the right source or report that the field cannot be established.
Blindly asking the model to “try again” can produce a different unsupported answer. Preserve the original candidate and failure reason, bound retries, and reject a repair that changes unrelated fields without evidence. Downstream writes should wait for the validation and approval appropriate to their consequences.
Choose acceptance cases that include absent values, contradictions, reversed relationships, wrong entities, and unrelated input. The most useful test is often a schema-valid object that your application must still refuse to act on.
Documentation and fixture date: October 28, 2026. Product documentation is a dated reference; this article makes no model-accuracy or production-readiness claim.
