InconclusiveCould not measure on claude-haiku-4-5Could not measure on gpt-5-miniCould not measure on gemini-3.1-flash-lite
Offering the model money
An offered tip increases output length. Length is the deterministic proxy this pilot can measure; the community claim is about quality, and the claim page will say so.
Offering the model a tip made answers longer. We could not measure whether they got better, which is what the claim is really about.
Why it matters
Length is easy to count and easy to mistake for quality.
Answer length rose by 44 to 180 words, but this metric has no threshold, so no verdict follows.
What the marks mean
- Could not measure
Show the per-model numbers
Could not be measured on claude-haiku-4-5, gpt-5-mini and gemini-3.1-flash-lite.
Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.
This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.
Ledger idC08-tip-length
What was tested
This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.
The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.
Result
| Measure | claude-haiku-4-5 | gpt-5-mini | gemini-3.1-flash-lite |
|---|---|---|---|
| Control arm | 204.1n 50 | 354.7n 50 | 518.5n 50 |
| Treatment arm | 248.2n 50 | 534.3n 50 | 575.0n 50 |
| Delta | +44.2 | +179.6 | +56.5 |
| Interval, 95 percent | 29.6 to 58.7 | 144.5 to 214.7 | 36.3 to 76.7 |
| Orbit | Unobservable100 of 100 records | Unobservable100 of 100 records | Unobservable100 of 100 records |
Orbit is assigned by the frozen status_v1 rule. This claim is scored on an word count, unbounded, which has no ceiling, so no threshold applies and no verdict is issued. The delta and its interval are published; the verdict is not.
Method for this claim
- Task set
- 10 matched open-ended prompts. The two arms are byte identical apart from the appended tip sentence.
- Runs per model per arm
- 5
- Scoring
- Deterministic, via scoreOutputLength. A committed function scores each answer with no model in the loop.
- Pass criterion as written for the pilot
- Reported as a mean word-count delta with an interval. No pass threshold is proposed for this claim in the pilot, because the measured quantity is length and the underlying community claim is about quality.
The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.
Model versions, as recorded
| Model | Version string returned | Delta on this claim | Interval |
|---|---|---|---|
| claude-haiku-4-5 | claude-haiku-4-5-20251001 | +44.2 | 29.6 to 58.7 |
| gpt-5-mini | gpt-5-mini-2025-08-07 | +179.6 | 144.5 to 214.7 |
| gemini-3.1-flash-lite | gemini-3.1-flash-lite | +56.5 | 36.3 to 76.7 |
Reading across models
Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.
gpt-5-mini rejected the fixed sampling setting and ran at its own default on all 1040 of its calls. The other models ran at temperature 0.