Holds
None against three examplesHelped a little, under our bar on claude-haiku-4-5Helped a little, under our bar on gpt-5-miniHelped a little, under our bar on gemini-3.1-flash-lite
None against one exampleHelped a little, under our bar on claude-haiku-4-5Helped a little, under our bar on gpt-5-miniHelped a little, under our bar on gemini-3.1-flash-lite
Showing examples
Few shot prompting: how many examples does a prompt need?
Give one good example of the answer you want. A second or third adds very little.
Why it matters
One example pins the shape of the answer. Writing more of them costs you time and buys almost nothing.
None against three examplesPass rate up 18 to 19 points on all three models.
None against one examplePass rate up 16 to 19 points on all three models.
What the marks mean
- Helped a little, under our bar
Show the per-model numbers
Claim tested: One worked example improves format compliance over no examples, on the same task.
Helped a little on claude-haiku-4-5, gpt-5-mini and gemini-3.1-flash-lite, under our bar.
Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.
This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.
Ledger idC04b-zero-vs-one
What was tested
This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.
The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.
Result
| Measure | claude-haiku-4-5 | gpt-5-mini | gemini-3.1-flash-lite | claude-haiku-4-5 | gpt-5-mini | gemini-3.1-flash-lite |
|---|---|---|---|---|---|---|
| Control arm | 0.814n 10 | 0.794n 10 | 0.800n 10 | 0.814n 10 | 0.783n 10 | 0.800n 10 |
| Treatment arm | 1.000n 10 | 0.977n 10 | 0.986n 10 | 0.971n 10 | 0.969n 10 | 0.957n 10 |
| Delta | +0.186 | +0.183 | +0.186 | +0.157 | +0.186 | +0.157 |
| Interval, 95 percent | 0.143 to 0.228 | 0.143 to 0.223 | 0.143 to 0.228 | 0.129 to 0.185 | 0.140 to 0.232 | 0.129 to 0.185 |
| Orbit | In free drift100 of 100 records | In free drift100 of 100 records | In free drift100 of 100 records | In free drift100 of 100 records | In free drift100 of 100 records | In free drift100 of 100 records |
Orbit is assigned by the frozen status_v1 rule. On this scale, deterministic pass rate, 0 to 1, the pass threshold is +0.20 and the failure floor is -0.20, each requiring an interval that excludes zero.
First attempt (retired)
An earlier task set tested the same question and was withdrawn: every model scored full marks with and without it, so the set could not separate the arms; a hardened set replaced it. The rows below are that first attempt, and they are kept because a set that measured nothing is evidence about the instrument rather than about the technique.
| Measure | claude-haiku-4-5 | gpt-5-mini | gemini-3.1-flash-lite |
|---|---|---|---|
| Control arm | 1.000n 50 | 1.000n 50 | 1.000n 50 |
| Treatment arm | 1.000n 50 | 1.000n 50 | 1.000n 50 |
| Delta | +0.000 | +0.000 | +0.000 |
| Interval, 95 percent | 0.000 to 0.000 | 0.000 to 0.000 | 0.000 to 0.000 |
| Orbit | Unobservable100 of 100 records | Unobservable100 of 100 records | Unobservable100 of 100 records |
Sets are now probed before they are funded, against a control-arm band that a saturated set cannot clear. How task sets are calibrated.
Method for this claim
- Task set
- 10 messy multi-sentence stock records reduced to one pipe-separated line of seven fields. Hardened a FOURTH time after calibration. v3 passed at 0.886 but all of its headroom sat on one field: the control wrote PT401 where the key wanted PT-401, because the instruction said uppercase-no-spaces and never mentioned the hyphen. The pair would have measured one hyphen rather than convention-pinning. v4 removes four MORE statements from the instruction, so the convention sites are: SKU punctuation and casing, date format, flag separator, and flag ordering. Each is unstated, each is legitimately ambiguous, and each is pinned by the examples. Headroom now distributes across fields.
- Runs per model per arm
- 50
- Scoring
- Deterministic, via scoreFieldwiseLine. A committed function scores each answer with no model in the loop.
- Pass criterion as written for the pilot
- Treatment mean fieldwise score exceeds control by at least 0.10 on the same 10 records.
The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.
Model versions, as recorded
| Model | Version string returned | Delta on this claim | Interval |
|---|---|---|---|
| claude-haiku-4-5 | claude-haiku-4-5-20251001 | +0.186 | 0.143 to 0.228 |
| gpt-5-mini | gpt-5-mini-2025-08-07 | +0.183 | 0.143 to 0.223 |
| gemini-3.1-flash-lite | gemini-3.1-flash-lite | +0.186 | 0.143 to 0.228 |
| claude-haiku-4-5 | claude-haiku-4-5-20251001 | +0.157 | 0.129 to 0.185 |
| gpt-5-mini | gpt-5-mini-2025-08-07 | +0.186 | 0.140 to 0.232 |
| gemini-3.1-flash-lite | gemini-3.1-flash-lite | +0.157 | 0.129 to 0.185 |
Reading across models
Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.
gpt-5-mini rejected the fixed sampling setting and ran at its own default on all 1040 of its calls. The other models ran at temperature 0.