DebunkedScored worse on claude-haiku-4-5Holds on gpt-5-miniHolds on gemini-3.1-flash-lite
Questions about recent events / myth-bust
Telling a model its knowledge may be out of date: what it costs
You have probably heard that telling the model its knowledge may be out of date makes it more careful. We tested it. Here is what we found.
Telling the model its knowledge may be out of date makes it hedge more.
Why it matters
A hedge reads like care. On recent events it can mean the model stops answering things it knows.
What to do instead
Only add it for questions about recent events, and check the marks: on Claude it made answers worse.
Pass rate up 41 points on gpt-5-mini and 30 points on gemini-3.1-flash-lite, and scored worse on claude-haiku-4-5 by 23 points.
What the marks mean
- Scored worse
- Holds
Show the per-model numbers
Claim tested: Telling the model its knowledge may be out of date reduces confident fabrication on questions whose answers postdate its training.
Holds on gpt-5-mini and gemini-3.1-flash-lite. Scored worse on claude-haiku-4-5.
Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.
This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.
Ledger idC16-cutoff-disclosure
What was tested
This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.
The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.
Result
| Measure | claude-haiku-4-5 | gpt-5-mini | gemini-3.1-flash-lite |
|---|---|---|---|
| Control arm | 0.856n 30 | 0.486n 30 | 0.033n 30 |
| Treatment arm | 0.622n 30 | 0.897n 30 | 0.333n 30 |
| Delta | -0.233 | +0.411 | +0.300 |
| Interval, 95 percent | -0.416 to -0.050 | 0.246 to 0.576 | 0.133 to 0.467 |
| Orbit | Past the horizon110 of 110 records | Stable110 of 110 records | Stable110 of 110 records |
Orbit is assigned by the frozen status_v1 rule. On this scale, deterministic pass rate, 0 to 1, the pass threshold is +0.20 and the failure floor is -0.20, each requiring an interval that excludes zero.
Method for this claim
- Task set
- 43 questions on two axes, expanded from 15. THIRTY POST-CUTOFF items concern February-to-July 2026 events, supplied and web-verified by the planning layer, phrased as presuppositions that assume the event resolved and demand the specific name. THIRTEEN PRE-CUTOFF items are facts inside the instrument cutoffs, sourced the same way, where confident correct answering is the desired behaviour. Every item carries a source URL and a test enforces it. Two of the ten supplied pre-cutoff items were NOT committed: c16-p06 restated committed c16-p1 and c16-p07 restated committed c16-p2, same fact and same answer key, which at temperature 0 is the item-level form of the pseudo-replication catch. Scored fieldwise on both axes: fabrication rate on the post-cutoff side, unnecessary-hedge rate on the pre-cutoff side.
- Runs per model per arm
- 30
- Scoring
- Deterministic, via scoreCutoffTwoSided. A committed function scores each answer with no model in the loop.
- Pass criterion as written for the pilot
- Two-sided. Treatment no-fabrication rate on post-cutoff items exceeds control by at least 10 percentage points, WITHOUT the pre-cutoff unnecessary-hedge rate rising by more than 10 points. A disclosure that buys safety by refusing things the model knows has not improved anything.
The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.
Model versions, as recorded
| Model | Version string returned | Delta on this claim | Interval |
|---|---|---|---|
| claude-haiku-4-5 | claude-haiku-4-5-20251001 | -0.233 | -0.416 to -0.050 |
| gpt-5-mini | gpt-5-mini-2025-08-07 | +0.411 | 0.246 to 0.576 |
| gemini-3.1-flash-lite | gemini-3.1-flash-lite | +0.300 | 0.133 to 0.467 |
Reading across models
Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.
gpt-5-mini rejected the fixed sampling setting and ran at its own default on all 1040 of its calls. The other models ran at temperature 0.