HoldsHolds on claude-haiku-4-5Holds on gpt-5-miniHolds on gemini-3.1-flash-lite
Getting the length right
Asking for an exact word count works. Vague length words do not
Ask for an exact word count. Vague words like short miss by a lot.
Why it matters
Short means one thing to you and another to the model. A number means the same thing to both.
How to use it
Say exactly how many words you want rather than keep it short. Expect the answer to land within a few percent of the number you asked for.
Pass rate up 68 to 95 points on all three models.
What the marks mean
- Holds
Show the per-model numbers
Claim tested: Asking for an exact word count gets you that word count.
Holds on claude-haiku-4-5, gpt-5-mini and gemini-3.1-flash-lite.
Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.
This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.
Ledger idC17-exact-length
What was tested
This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.
The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.
Result
| Measure | claude-haiku-4-5 | gpt-5-mini | gemini-3.1-flash-lite |
|---|---|---|---|
| Control arm | 0.247n 10 | 0.016n 10 | 0.310n 10 |
| Treatment arm | 0.948n 10 | 0.968n 10 | 0.994n 10 |
| Delta | +0.701 | +0.952 | +0.684 |
| Interval, 95 percent | 0.476 to 0.926 | 0.920 to 0.984 | 0.429 to 0.939 |
| Orbit | Stable100 of 100 records | Stable100 of 100 records | Stable100 of 100 records |
Orbit is assigned by the frozen status_v1 rule. On this scale, deterministic pass rate, 0 to 1, the pass threshold is +0.20 and the failure floor is -0.20, each requiring an interval that excludes zero.
Method for this claim
- Task set
- 10 generation tasks at three target lengths (25, 100, 250 words). The control arm asks in the vague register people actually use (short, a few paragraphs, a detailed piece); the treatment arm asks for exactly N words. Both arms are measured against the SAME target, so the control is not being asked to guess a number it was never given: it is being measured on whether the vague register lands anywhere near the length the asker had in mind.
- Runs per model per arm
- 50
- Scoring
- Deterministic, via scoreWordCountCloseness. A committed function scores each answer with no model in the loop.
- Pass criterion as written for the pilot
- Reported as a measured deviation rather than a pass or fail. The claim holds if treatment mean closeness exceeds control by at least 0.10 and the within-5-percent band rate rises.
The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.
Model versions, as recorded
| Model | Version string returned | Delta on this claim | Interval |
|---|---|---|---|
| claude-haiku-4-5 | claude-haiku-4-5-20251001 | +0.701 | 0.476 to 0.926 |
| gpt-5-mini | gpt-5-mini-2025-08-07 | +0.952 | 0.920 to 0.984 |
| gemini-3.1-flash-lite | gemini-3.1-flash-lite | +0.684 | 0.429 to 0.939 |
Reading across models
Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.
gpt-5-mini rejected the fixed sampling setting and ran at its own default on all 1040 of its calls. The other models ran at temperature 0.