Prompt, then revise / myth-bust
Does iterative prompting help?
You have probably heard that asking the model to critique and revise its own draft improves it. We tested it. Here is what we found.
Asking the model to revise its own draft did not make it better.
Why it matters
The revise pass costs you a whole extra turn. In our tests it bought nothing back.
What to do instead
Write one careful prompt. Revise only when you can name what is wrong.
Debunked
Measured on two models; the rest could not be measured on this task set. Those results are tested via API. Also measured in Claude Code, reported separately on this page and never averaged with this.
All 4 models tested already passed this without the change, and still passed with it.
There is no chart here because there is no shape to draw: every value sits at one end of the scale, the models agree, and no range of likely values is wide enough to see. The per-model numbers are in the tables below.
tested via API · Could not measure on Claude Haiku 4.5, tested via API · No measured effect on GPT-5 mini, tested via API · No measured effect on Gemini 3.1 Flash Lite, tested via API
tested in Claude Code · No measured effect on Claude Haiku 4.5, multi-turn, tested in Claude Code
What the marks mean
- Could not measure
- No measured effect
Show per-task detail
Without to with, per model
Via API tested via API
GPT-5 mini
No measured effect
Gemini 3.1 Flash Lite
No measured effect
Claude Haiku 4.5
Could not measure
deterministic pass rate, 0 to 1
In Claude Code tested in Claude Code
Claude Haiku 4.5 multi-turn
No measured effect
deterministic pass rate, 0 to 1
The Claude Code readings on this page are the multi-turn run, which committed its comparison as arm means and an interval rather than as per-item values. A profile of the items behind that interval cannot be drawn from what it commits, so this page keeps the without-to-with chart.
Result
A generic refine pass after the answer could be measured on two of the three models tested via API, so no reading covers the set, and showed no measured effect on Claude Haiku 4.5 via the multi-turn run in Claude Code. Measured 2026-08-13.
Show the per-model numbers
Claim tested: A generic refine pass after the answer beats one careful prompt.
No measured effect on GPT-5 mini and Gemini 3.1 Flash Lite. Could not be measured on Claude Haiku 4.5.
Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.
This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.
Ledger idC15-iterative-refinement
Correction: the runs figure on this page was the run plan, not a count
Until 2026-08-31 this page stated 50 runs per model per arm. That figure was the claim record's declared run plan, which the harness reads to decide how many calls to make. It was never a count of anything.
The records behind this page hold 10 records per model per arm, and the figure it publishes now is "1 pass x 10 items", counted from them.
No verdict, delta, interval or coverage figure changes. They never read the declared plan: each cell is built from the records themselves, which is why the wrong figure could sit beside correct results for as long as it did.
Dated . Corrections on this site are appended and never rewritten.
What was tested
This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.
The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.
Per-model numbers
| Measure | Claude Haiku 4.5 | GPT-5 mini | Gemini 3.1 Flash Lite |
|---|---|---|---|
| Control arm | 1.000n 10 | 0.971n 10 | 0.971n 10 |
| Treatment arm | 1.000n 10 | 1.000n 10 | 0.971n 10 |
| Delta | +0.000 | +0.029 | +0.000 |
| Interval, 95 percent | 0.000 to 0.000 | -0.009 to 0.066 | 0.000 to 0.000 |
| Orbit | Unobservable20 of 20 records | In free drift20 of 20 records | In free drift20 of 20 records |
Orbit is assigned by the frozen status_v1 rule. On this scale, deterministic pass rate, 0 to 1, the pass threshold is +0.20 and the failure floor is -0.20, each requiring an interval that excludes zero.
In Claude Code
These cells are tested in Claude Code, on a subscription path with no API key. They are a second instrument and are never averaged with the figures above, which are tested via API. What that means, and how it was calibrated.
| Model | Control | Treatment | Delta | Interval | Pairs | Orbit |
|---|---|---|---|---|---|---|
| Claude Haiku 4.5 | 0.9571 | 0.9857 | +0.029 | -0.041 to 0.098 | 10 | In free drift |
The multi-turn run recorded intervals and a continuity comparison rather than a verdict. The Orbit above is assigned from those committed figures by the same status_v1 rule used everywhere else on this site; no threshold or interval is recomputed.
| Model | Control | Treatment | Delta | Interval | Pairs | Orbit |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 1.0000 | 1.0000 | +0.000 | 0.000 to 0.000 | 10 | Unobservable |
| Claude Sonnet 5 | 0.9857 | 0.9429 | -0.043 | -0.127 to 0.041 | 10 | In free drift |
| Claude Fable 5.1 | 1.0000 | 0.9857 | -0.014 | -0.042 to 0.014 | 10 | In free drift |
These cells have no API counterpart. The wave 2 API records cover this claim on Claude Haiku 4.5 and on no other model, so nothing above is compared against an API interval and no continuity reading is taken from it: an overlap against a different model’s API cell would read a difference between models as a difference between instruments. The panel holds the instrument fixed and varies the model. The transport verdict is unchanged.
No row here says one model is better than another, and none attributes a difference to a cause: no model on this panel was run twice, so a cell that differs and a re-run of the same model are not distinguishable by this design. The Orbit is assigned from the committed figures by the same status_v1 rule used everywhere else on this site; no threshold or interval is recomputed.
| Model | Control | Treatment | Delta | Interval | Pairs | Orbit |
|---|---|---|---|---|---|---|
| Claude Haiku 4.5 (not on the panel) | 0.9571 | 0.9857 | +0.029 | -0.041 to 0.098 | 10 | In free drift |
| Claude Fable 5.1 (earlier pass) | 1.0000 | 1.0000 | +0.000 | 0.000 to 0.000 | 10 | Unobservable |
The earlier pass is the same model measured on the same instrument under an older gate, before records carried a schema version, an invocation id or the served-model rule. It is a second reading, not an average: two passes of one model are two numbers, and averaging them would hide whichever of the two moved, which is the only thing a repeat pass is good for.
The Claude Code interval on this claim overlaps its API interval. Across the multi-turn run the pre-registered comparison returned "continuous with panel v1, 3 of 3 cells", and that statement covers the cells it measured and is not extrapolated to any cell that was not run.
Method for this claim
- Task set
- 10 generation tasks each carrying EIGHT mechanically checkable requirements. Hardened after calibration returned 1.000: five requirements were satisfied first time on every task, so a refine pass had nothing to fix. v2 adds three more per task and tightens the word caps, so the control has a realistic chance of dropping one.
- Runs per model per arm
- 1 pass x 10 items
- Scoring
- Deterministic, via scoreRequirementKeys. A committed function scores each answer with no model in the loop.
- Pass criterion as written for the pilot
- Two-sided. Treatment mean requirement satisfaction exceeds control by at least 0.10, OR the regression rate is material enough to publish as a cost. A refine pass that neither helps nor harms is a null and is reported as one.
The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.
Model versions, as recorded
| Model | Version string returned | Delta on this claim | Interval |
|---|---|---|---|
| Claude Haiku 4.5 | claude-haiku-4-5-20251001 | +0.000 | 0.000 to 0.000 |
| GPT-5 mini | gpt-5-mini-2025-08-07 | +0.029 | -0.009 to 0.066 |
| Gemini 3.1 Flash Lite | gemini-3.1-flash-lite | +0.000 | 0.000 to 0.000 |
Reading across models
Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.
GPT-5 mini rejected the fixed sampling setting and ran at its own default on all 1,040 of its calls. The other models ran at temperature 0.