Getting clean JSON back
Does a JSON schema help?
Show the model the exact shape you want back. Write out the field names instead of just asking for JSON.
Why it matters
Without a shape to copy, the model picks its own field names and your code breaks.
How to use it
Put a small example of the structure in your prompt. List every field you need and what kind of value it holds.
Holds
Pass rate rose from 0% to 100% on all three models. Those results are tested via API. Also measured in Claude Code and Codex, reported separately on this page and never averaged with this.
All 8 models tested went from failing this to passing it.
There is no chart here because there is no shape to draw: every value sits at one end of the scale, the models agree, and no range of likely values is wide enough to see. The per-model numbers are in the tables below.
tested via API · Holds on Claude Haiku 4.5, tested via API · Holds on GPT-5 mini, tested via API · Holds on Gemini 3.1 Flash Lite, tested via API
tested in Claude Code · Holds on Claude Fable 5, tested in Claude Code · Holds on Claude Opus 5, tested in Claude Code · Holds on Claude Haiku 4.5, tested in Claude Code · Holds on Claude Sonnet 5, tested in Claude Code · Holds on Claude Fable 5.1, tested in Claude Code
tested in Codex · Holds on GPT-5.4 mini, Codex run on GPT-5.4 mini, tested in Codex · Holds on GPT-5.6 Luna, Codex run on GPT-5.6 Luna, tested in Codex · Holds on GPT-5.6 Terra, Codex run on GPT-5.6 Terra, tested in Codex
What the marks mean
- Holds
Show per-task detail
Every item, without and with, per model
Via API tested via API
- Claude Haiku 4.5
- Gemini 3.1 Flash Lite
- GPT-5 mini
deterministic pass rate, 0 to 1
Hover or focus an item to read its task and every model’s two values.
Pooled: the interval is taken over the records, not these means. These are those same records collapsed per item, so where an item carries more passes than another the lines need not average to the figure the table prints.
The item means behind this plot
| Item | Claude Haiku 4.5 | Gemini 3.1 Flash Lite | GPT-5 mini | |||
|---|---|---|---|---|---|---|
| Task | without | with | without | with | without | with |
| t01 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t02 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t03 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t04 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t05 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t06 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t07 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t08 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t09 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t10 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
In Claude Code tested in Claude Code
- Claude Fable 5
- Claude Fable 5.1
- Claude Haiku 4.5
- Claude Opus 5
- Claude Sonnet 5
deterministic pass rate, 0 to 1
Hover or focus an item to read its task and every model’s two values.
Pooled: the interval is taken over the records, not these means. These are those same records collapsed per item, so where an item carries more passes than another the lines need not average to the figure the table prints.
The item means behind this plot
| Item | Claude Fable 5 | Claude Fable 5.1 | Claude Haiku 4.5 | Claude Opus 5 | Claude Sonnet 5 | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Task | without | with | without | with | without | with | without | with | without | with |
| t01 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t02 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t03 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t04 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t05 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t06 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t07 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t08 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t09 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| t10 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
Result
An explicit JSON schema in the prompt raised the pass rate from 0% to 100% on all three models tested via API, and held on all five models tested in Claude Code, and held on GPT-5.4 mini via the Codex run on GPT-5.4 mini, and held on GPT-5.6 Luna via the Codex run on GPT-5.6 Luna, and held on GPT-5.6 Terra via the Codex run on GPT-5.6 Terra. Measured 2026-08-12.
Show the per-model numbers
Claim tested: An explicit JSON schema in the prompt yields a higher valid-JSON rate than an unstructured instruction to respond in JSON.
Holds on Claude Haiku 4.5, GPT-5 mini and Gemini 3.1 Flash Lite.
Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.
This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.
Tier 1 and tier 2
On tier 1, the side with the tip already scored near the top for every model. Tier 2 is a harder set of ten tasks, run once on each side, to see if the tip still helps when the task is hard enough to fail. Each tier is its own result. The two are never averaged.
The tier 1 interval here pools every answer. The tier 2 interval pairs the two sides task by task.
| Model | Tier 1 (the tables below) | Tier 2 (the harder set) | ||||
|---|---|---|---|---|---|---|
| Result | Without, then with the tip | Change (95% interval) | Result | Without, then with the tip | Change (95% interval) | |
| Tested in Claude Code | ||||||
| Claude Opus 5 | Holds | 0% to 100% | +100 points (+100 to +100) | Holds | 0% to 100% | +100 points (+100 to +100) |
| Claude Sonnet 5 | Holds | 0% to 100% | +100 points (+100 to +100) | Holds | 0% to 80% | +80 points (+54 to +106) |
| Claude Haiku 4.5 | Holds | 0% to 100% | +100 points (+100 to +100) | Holds | 0% to 100% | +100 points (+100 to +100) |
| Claude Fable 5.1 | Holds | 0% to 100% | +100 points (+100 to +100) | Holds | 0% to 100% | +100 points (+100 to +100) |
| Tested via the API | ||||||
| GPT-5 mini | Holds | 0% to 100% | +100 points (+100 to +100) | Holds | 0% to 100% | +100 points (+100 to +100) |
| Gemini 3.1 Flash Lite | Holds | 0% to 100% | +100 points (+100 to +100) | Holds | 0% to 100% | +100 points (+100 to +100) |
Ledger idC01-json-schema
Clarification, . Until 2026-08-31 this figure read 5, which was the pass count rather than a number of calls. What changed, and why it is filed once.
What was tested
This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.
The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.
Per-model numbers
| Measure | Claude Haiku 4.5 | GPT-5 mini | Gemini 3.1 Flash Lite |
|---|---|---|---|
| Control arm | 0.000n 50 | 0.000n 50 | 0.000n 50 |
| Treatment arm | 1.000n 50 | 1.000n 50 | 1.000n 50 |
| Delta | +1.000 | +1.000 | +1.000 |
| Interval, 95 percent | 1.000 to 1.000 | 1.000 to 1.000 | 1.000 to 1.000 |
| Orbit | Stable100 of 100 records | Stable100 of 100 records | Stable100 of 100 records |
| Same answer on a repeat | 20 of 20repeated tasksat temperature 0 | 20 of 20repeated tasksat the vendor default | 20 of 20repeated tasksat temperature 0 |
- Claude Haiku 4.5. It gave the same answer on 20 of 20 repeated tasks. Those answers were held at temperature 0.
- GPT-5 mini. It gave the same answer on 20 of 20 repeated tasks. Those answers were at the vendor's own default setting, because it refused temperature 0.
- Gemini 3.1 Flash Lite. It gave the same answer on 20 of 20 repeated tasks. Those answers were held at temperature 0.
A separate run asked each of these tasks five times, with no tip applied. Its figures are not part of the table above.
- GPT-5 mini. Asked each task five times, it gave the same answer all five times on 10 of 10. The likely range for that share is 72 to 100 percent. Those answers were at the vendor's own default setting, because it refused temperature 0.
- Gemini 3.1 Flash Lite. Asked each task five times, it gave the same answer all five times on 10 of 10. The likely range for that share is 72 to 100 percent. Those answers were held at temperature 0.
Orbit is assigned by the frozen status_v1 rule. On this scale, deterministic pass rate, 0 to 1, the pass threshold is +0.20 and the failure floor is -0.20, each requiring an interval that excludes zero.
In Claude Code
These cells are tested in Claude Code, on a subscription path with no API key. They are a second instrument and are never averaged with the figures above, which are tested via API. What that means, and how it was calibrated.
| Model | Control | Treatment | Delta | Interval | Pairs | Orbit |
|---|---|---|---|---|---|---|
| Claude Fable 5 | 0.0000 | 1.0000 | +1.000 | 1.000 to 1.000 | 50 | Stable |
| Claude Opus 5 | 0.0000 | 1.0000 | +1.000 | 1.000 to 1.000 | 50 | Stable |
| Claude Haiku 4.5 | 0.0000 | 1.0000 | +1.000 | 1.000 to 1.000 | 50 | Stable |
| Claude Sonnet 5 | 0.0000 | 1.0000 | +1.000 | 1.000 to 1.000 | 50 | Stable |
| Claude Fable 5.1 | 0.0000 | 1.0000 | +1.000 | 1.000 to 1.000 | 50 | Stable |
Two separate runs asked each of these tasks five times, with no tip applied. Claude Fable 5 was asked in the later one, the same way. Their figures are not part of the table above.
- Claude Fable 5. Asked each task five times, it gave the same answer all five times on 10 of 10. The likely range for that share is 72 to 100 percent. Those answers were with no sampling control, which this way of testing does not offer.
- Claude Opus 5. Asked each task five times, it gave the same answer all five times on 10 of 10. The likely range for that share is 72 to 100 percent. Those answers were with no sampling control, which this way of testing does not offer. Added 25 September 2026, as a report and not a finding. Outside reports say an unreleased Opus 5.2 answered some requests sent to Opus 5 between 14 and 17 September. Every answer in this run was made on 17 September. Each one names Opus 5 as the model that answered. That name would not have told the two apart, so this run cannot say whether it was affected.
- Claude Haiku 4.5. Asked each task five times, it gave the same answer all five times on 10 of 10. The likely range for that share is 72 to 100 percent. Those answers were with no sampling control, which this way of testing does not offer.
- Claude Sonnet 5. Asked each task five times, it gave the same answer all five times on 10 of 10. The likely range for that share is 72 to 100 percent. Those answers were with no sampling control, which this way of testing does not offer.
- Claude Fable 5.1. Asked each task five times, it gave the same answer all five times on 10 of 10. The likely range for that share is 72 to 100 percent. Those answers were with no sampling control, which this way of testing does not offer.
| Model | Control | Treatment | Delta | Interval | Pairs | Orbit |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 0.0000 | 1.0000 | +1.000 | 1.000 to 1.000 | 10 | Stable |
Method for this claim
- Task set
- 10 short biographical snippets, varied in sentence order, length, and how directly each field is stated. Both arms ask for the same three fields; only the schema statement differs. Hardened for instrument v2: v1 asked for three obvious fields (name, role, city) that any model emits under a bare respond-in-JSON instruction. v2 requires five fields whose key names are not guessable from the prose, so an unstructured arm produces valid JSON with the wrong keys.
- Runs per model per arm
- 5 passes x 10 items
- Scoring
- Deterministic, via scoreJsonValidAndShaped. A committed function scores each answer with no model in the loop.
- Pass criterion as written for the pilot
- Treatment valid-JSON rate exceeds control by at least 10 percentage points on the same 10 inputs at the same model and settings.
The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.
Model versions, as recorded
| Model | Version string returned | Delta on this claim | Interval |
|---|---|---|---|
| Claude Haiku 4.5 | claude-haiku-4-5-20251001 | +1.000 | 1.000 to 1.000 |
| GPT-5 mini | gpt-5-mini-2025-08-07 | +1.000 | 1.000 to 1.000 |
| Gemini 3.1 Flash Lite | gemini-3.1-flash-lite | +1.000 | 1.000 to 1.000 |
Reading across models
Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.
GPT-5 mini rejected the fixed sampling setting and ran at its own default on all 1,040 of its calls. The other models ran at temperature 0.