DebunkedCould not measure on claude-haiku-4-5No measured effect on gpt-5-miniNo measured effect on gemini-3.1-flash-lite
Prompt, then revise / myth-bust
Iterative prompting: does asking for a revision improve the answer?
You have probably heard that asking the model to critique and revise its own draft improves it. We tested it. Here is what we found.
Asking the model to revise its own draft did not make it better.
Why it matters
The revise pass costs you a whole extra turn. In our tests it bought nothing back.
What to do instead
Write one careful prompt. Revise only when you can name what is wrong.
Measured on two models; the rest could not be measured on this task set.
What the marks mean
- Could not measure
- No measured effect
Show the per-model numbers
Claim tested: A generic refine pass after the answer beats one careful prompt.
No measured effect on gpt-5-mini and gemini-3.1-flash-lite. Could not be measured on claude-haiku-4-5.
Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.
This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.
Ledger idC15-iterative-refinement
What was tested
This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.
The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.
Result
| Measure | claude-haiku-4-5 | gpt-5-mini | gemini-3.1-flash-lite |
|---|---|---|---|
| Control arm | 1.000n 10 | 0.971n 10 | 0.971n 10 |
| Treatment arm | 1.000n 10 | 1.000n 10 | 0.971n 10 |
| Delta | +0.000 | +0.029 | +0.000 |
| Interval, 95 percent | 0.000 to 0.000 | -0.009 to 0.066 | 0.000 to 0.000 |
| Orbit | Unobservable20 of 20 records | In free drift20 of 20 records | In free drift20 of 20 records |
Orbit is assigned by the frozen status_v1 rule. On this scale, deterministic pass rate, 0 to 1, the pass threshold is +0.20 and the failure floor is -0.20, each requiring an interval that excludes zero.
Method for this claim
- Task set
- 10 generation tasks each carrying EIGHT mechanically checkable requirements. Hardened after calibration returned 1.000: five requirements were satisfied first time on every task, so a refine pass had nothing to fix. v2 adds three more per task and tightens the word caps, so the control has a realistic chance of dropping one.
- Runs per model per arm
- 50
- Scoring
- Deterministic, via scoreRequirementKeys. A committed function scores each answer with no model in the loop.
- Pass criterion as written for the pilot
- Two-sided. Treatment mean requirement satisfaction exceeds control by at least 0.10, OR the regression rate is material enough to publish as a cost. A refine pass that neither helps nor harms is a null and is reported as one.
The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.
Model versions, as recorded
| Model | Version string returned | Delta on this claim | Interval |
|---|---|---|---|
| claude-haiku-4-5 | claude-haiku-4-5-20251001 | +0.000 | 0.000 to 0.000 |
| gpt-5-mini | gpt-5-mini-2025-08-07 | +0.029 | -0.009 to 0.066 |
| gemini-3.1-flash-lite | gemini-3.1-flash-lite | +0.000 | 0.000 to 0.000 |
Reading across models
Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.
gpt-5-mini rejected the fixed sampling setting and ran at its own default on all 1040 of its calls. The other models ran at temperature 0.