Model under test
Claude Fable 5.1
Every result this site holds for Claude Fable 5.1. We tested it inside Claude Code.
Version string recorded, as returned: claude-fable-5-1. The answers carry the requested name back, with no dated snapshot behind it.
What was measured, and where
We ran 45 separate tests on this model, listed by test method below and never added together. How the test methods were compared.
Prompting tips, replayed in Claude Code
12 tests, tested in Claude Code, Single-turn replay.
- Could not measure 5 of 12
- No measured effect 4 of 12
- Helped 3 of 12
Skills, in Claude Code: Fable 5.1, single attempt
8 tests, tested in Claude Code, Fable 5.1, single attempt.
- No measured effect 6 of 8
- Scored worse 1 of 8
- Could not measure 1 of 8
Skills, in Claude Code: Wider run, three ways
25 tests, tested in Claude Code, Wider run, three ways.
- No measured effect 22 of 25
- Could not measure 2 of 25
- Scored worse 1 of 25
What these figures can be set beside
Every table on this page is one job, run one way, on one occasion. A model tested the other way is not on those tables and its figures are not set against these. We checked the two test methods against each other and they came out different, so they are reported side by side and never averaged.
- Measured on
- tested in Claude Code.
- Not directly comparable
- Gemini 3.1 Flash Lite; GPT-5 mini; GPT-5.4 mini were tested through the API, which this model did not run. No figure below is theirs and none of theirs is set against these.
- Not directly comparable
- GPT-5.4 mini; GPT-5.6 Luna; GPT-5.6 Terra; GPT-6 Luna; GPT-6 Sol were tested inside Codex, which this model did not run. No figure below is theirs and none of theirs is set against these.
The model measured on both, job by job
Claude Haiku 4.5 was tested through the API and inside Claude Code. Its readings are the only thing connecting them, and they are stated job by job, never combined into one figure. It is one of two models that connect a pair of test methods this way. Each connects its own pair, and no chain is drawn through them.
Show the readings, job by job (17 jobs)
- Accessibility audit Claude Haiku 4.5 scored 40.0% (26.3% to 53.7%) tested via API, First run, with and without the skill, and 45.5% (35.0% to 56.0%) tested in Claude Code, Second run, every task twice, and 46.1% (34.1% to 58.1%) tested in Claude Code, Wider run, three ways. They are stated side by side and are not differenced. How well the two paths agree here: 8 of 8 tests agree. What was compared.
- Code review Claude Haiku 4.5 scored 58.3% (47.8% to 68.7%) tested in Claude Code, Wider run, three ways. That is one test method only, so it bridges nothing on its own.
- On-page audit Claude Haiku 4.5 scored 23.9% (1.8% to 46.1%) tested via API, First run, with and without the skill, and 23.1% (6.6% to 39.6%) tested in Claude Code, Second run, every task twice, and 23.1% (3.6% to 42.6%) tested in Claude Code, Wider run, three ways. They are stated side by side and are not differenced. How well the two paths agree here: 8 of 8 tests agree. What was compared.
- Skill authoring Claude Haiku 4.5 scored 61.8% (52.3% to 71.3%) tested via API, First run, with and without the skill, and 70.0% (63.5% to 76.5%) tested in Claude Code, Second run, every task twice, and 67.3% (59.7% to 74.9%) tested in Claude Code, Wider run, three ways. They are stated side by side and are not differenced. How well the two paths agree here: 8 of 8 tests agree. What was compared.
- Spec writing Claude Haiku 4.5 scored 99.5% (98.7% to 100.0%†) tested via API, First run, with and without the skill, and 98.6% (97.7% to 99.6%) tested in Claude Code, Second run, every task twice, and 99.2% (98.7% to 99.7%) tested in Claude Code, Wider run, three ways. They are stated side by side and are not differenced. How well the two paths agree here: 8 of 8 tests agree. What was compared.
- Getting clean JSON back Claude Haiku 4.5 scored 0.0% (0.0% to 0.0%) tested via API, Main run, tip by tip, and 0.0% (0.0% to 0.0%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Instructions before or after Claude Haiku 4.5 scored 100.0% (100.0% to 100.0%) tested via API, Main run, tip by tip, and 100.0% (100.0% to 100.0%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Using tags and formatting Claude Haiku 4.5 scored 100.0% (100.0% to 100.0%) tested via API, Main run, tip by tip, and 100.0% (100.0% to 100.0%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Showing examples Claude Haiku 4.5 scored 100.0% (100.0% to 100.0%) tested in Claude Code. That is one test method only, so it bridges nothing on its own.
- Showing examples Claude Haiku 4.5 scored 81.4% (77.2% to 85.7%) tested via API, Main run, tip by tip, and 81.4% (77.2% to 85.7%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Showing examples Claude Haiku 4.5 scored 81.4% (77.2% to 85.7%) tested via API, Main run, tip by tip, and 81.4% (77.2% to 85.7%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Better step-by-step answers Claude Haiku 4.5 scored 30.0% (17.2% to 42.8%) tested via API, Main run, tip by tip, and 20.0% (8.8% to 31.2%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Stop made-up answers Claude Haiku 4.5 scored 0.0% (0.0% to 0.0%) tested via API, Main run, tip by tip, and 22.0% (10.4% to 33.6%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Instructions in long prompts Claude Haiku 4.5 scored 100.0% (100.0% to 100.0%) tested via API, Main run, tip by tip, and 100.0% (100.0% to 100.0%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Offering the model money Claude Haiku 4.5 scored 204 words (199 words to 209 words) tested via API, Main run, tip by tip, and 241 words (236 words to 247 words) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Questions about recent events Claude Haiku 4.5 scored 85.6% (73.2% to 98.0%) tested via API, Main run, tip by tip, and 96.7% (90.1% to 100.0%†) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
- Getting the length right Claude Haiku 4.5 scored 24.7% (1.2% to 48.2%) tested via API, Main run, tip by tip, and 27.5% (0.0% to 54.9%) tested in Claude Code. They are stated side by side and are not differenced. How well the two paths agree here: second test method, 1 of 2 tests. What was compared.
† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.
- The grader, separately
- a second test method, 92 of 120 picks, 10 of 12 tests. The grader itself, replayed on the committed judgements rather than on any subject. Both criteria we set in advance failed, and the graded classes stayed unrun on this path because of it.
What this model does on its own, one job at a time
We measured this model on 17 kinds of work, and the scores for every one of them are below.
Show the figures, one job at a time
One table per job and per test method, ranked by what the model did with no help. Nothing here is averaged across a job, a test method or a run. Every model’s tables sit together on the comparison page.
Claude Fable 5.1 scored higher than all 4 other models measured on this work.
What one task costs with no help, and what it scores Stop made-up answers
- tested via API
- tested in Claude Code
Across: what one task costs with no help, in US dollars at list price. Each gridline is ten times the one before it. Up: deterministic pass rate, 0 to 1.
- Gemini 3.1 Flash Lite Main run, tip by tip tested via API $0.00019 per task scored 0.0%, 0.0% to 0.0%
- GPT-5 mini Main run, tip by tip tested via API $0.00039 per task scored 20.0%, 8.8% to 31.2%
- Claude Haiku 4.5 Main run, tip by tip tested via API $0.00161 per task scored 0.0%, 0.0% to 0.0%
- Claude Haiku 4.5 tested in Claude Code $0.00340 per task scored 22.0%, 10.4% to 33.6%
- Claude Opus 5.5 Opus 5.5 series, each task asked once tested in Claude Code $0.00382 per task scored 90.0%, 70.4% to 100.0%† 12.2 s per call, middle half 12.0 s to 13.0 s
- Claude Sonnet 5 tested in Claude Code $0.01083 per task scored 38.0%, 24.4% to 51.6%
- Claude Opus 5 tested in Claude Code $0.02363 per task scored 30.0%, 17.2% to 42.8%
- Claude Fable 5.1 tested in Claude Code $0.03175 per task scored 76.0%, 64.0% to 88.0%
- Claude Fable 5 tested in Claude Code $0.03501 per task scored 24.0%, 12.0% to 36.0%
- † The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.
- Marks here are all one size. Not every mark has a time behind it. Sizing only the ones that do would draw the rest at a size we did not measure. Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Measured, and not on this chart
- GPT-5.4 mini Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
- GPT-5.6 Luna Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
- GPT-5.6 Terra Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
- GPT-6 Luna Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
- GPT-6 Sol Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
This is one of 17 sets of work this model was measured on. The same chart for every one of them sits on the comparison page, and the tables below carry this model’s figures for all of them.
Accessibility audit · seeded accessibility defects
tested in Claude Code, Fable 5.1, single attempt
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 85.8% 72.6% to 99.1% 20 records over 10 items, pooled from 2 entries | 93.5% rampstackco-accessibility-audit (of 2 entries) | $0.05119 run on subscription; shown at API list price for comparison | 948 in, 834 out | 755 | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Accessibility audit · seeded accessibility defects
tested in Claude Code, Wider run, three ways
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 87.9% 76.3% to 99.5% 40 records over 10 items, pooled from 4 entries | 91.5% alirezarezvani-a11y-audit (of 4 entries) | $0.05126 run on subscription; shown at API list price for comparison | 949 in, 835 out | 755 | one pass, not measured | not measured |
| 2 | Claude Opus 5 | 87.1% 78.0% to 96.2% 40 records over 10 items, pooled from 4 entries | 91.5% affaan-m-accessibility (of 4 entries) | $0.00704 run on subscription; shown at API list price for comparison | 946 in, 92 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 3 | Claude Sonnet 5 | 80.2% 71.4% to 88.9% 40 records over 10 items, pooled from 4 entries | 80.7% alirezarezvani-a11y-audit (of 4 entries) | $0.00368 run on subscription; shown at API list price for comparison | 1,272 in, 114 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 4 | Claude Haiku 4.5 | 46.1% 34.1% to 58.1% 40 records over 10 items, pooled from 4 entries | 62.8% rampstackco-accessibility-audit (of 4 entries) | $0.00104 run on subscription; shown at API list price for comparison | 765 in, 56 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Code review · seeded code defects
tested in Claude Code, Wider run, three ways
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5 | 93.5% 84.7% to 100.0%† 80 records over 10 items, pooled from 8 entries | 95.5% addyosmani-code-review-and-quality (of 8 entries) | $0.00948 run on subscription; shown at API list price for comparison | 1,333 in, 113 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 2 | Claude Fable 5.1 | 89.0% 83.2% to 94.8% 80 records over 10 items, pooled from 8 entries | 93.5% tied: jeffallan-code-reviewer, mattpocock-code-review, open-gsd-code-review (of 8 entries) | $0.04548 run on subscription; shown at API list price for comparison | 1,349 in, 640 out | 555 | one pass, not measured | not measured |
| 3 | Claude Sonnet 5 | 74.9% 62.4% to 87.4% 80 records over 10 items, pooled from 8 entries | 83.3% rampstackco-code-review-web (of 8 entries) | $0.00406 run on subscription; shown at API list price for comparison | 1,659 in, 74 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 4 | Claude Haiku 4.5 | 58.3% 47.8% to 68.7% 80 records over 10 items, pooled from 8 entries | 56.7% open-gsd-code-review (of 8 entries) | $0.00120 run on subscription; shown at API list price for comparison | 998 in, 41 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
On-page audit · seeded on-page issues
tested in Claude Code, Fable 5.1, single attempt
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 98.3% 95.1% to 100.0%† 20 records over 10 items, pooled from 2 entries | 95.0% tied: affaan-m-seo, rampstackco-seo-onpage (of 2 entries) | $0.03907 run on subscription; shown at API list price for comparison | 1,070 in, 568 out | 499 | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
On-page audit · seeded on-page issues
tested in Claude Code, Wider run, three ways
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 97.2% 93.5% to 100.0%† 30 records over 10 items, pooled from 3 entries | 98.3% rampstackco-seo-onpage (of 3 entries) | $0.03873 run on subscription; shown at API list price for comparison | 1,070 in, 561 out | 491 | one pass, not measured | not measured |
| 2 | Claude Opus 5 | 88.1% 76.9% to 99.2% 30 records over 10 items, pooled from 3 entries | 83.3% alirezarezvani-seo-audit (of 3 entries) | $0.00715 run on subscription; shown at API list price for comparison | 1,068 in, 72 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 3 | Claude Sonnet 5 | 53.0% 43.8% to 62.2% 30 records over 10 items, pooled from 3 entries | 66.5% affaan-m-seo (of 3 entries) | $0.00362 run on subscription; shown at API list price for comparison | 1,394 in, 84 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 4 | Claude Haiku 4.5 | 23.1% 3.6% to 42.6% 30 records over 10 items, pooled from 3 entries | 41.1% rampstackco-seo-onpage (of 3 entries) | $0.00105 run on subscription; shown at API list price for comparison | 832 in, 43 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Skill authoring · specification requirements
tested in Claude Code, Fable 5.1, single attempt
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 100.0% 100.0% to 100.0% 20 records over 10 items, pooled from 2 entries | 100.0% rampstackco-skill-creation-walkthrough (of 2 entries) | $0.15512 run on subscription; shown at API list price for comparison | 315 in, 3,039 out | 145 | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Skill authoring · specification requirements
tested in Claude Code, Wider run, three ways
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 100.0% 100.0% to 100.0% 40 records over 10 items, pooled from 4 entries | 100.0% tied: openclaw-skill-creator, yusufkaraaslan-skill-builder (of 4 entries) | $0.16037 run on subscription; shown at API list price for comparison | 316 in, 3,144 out | 137 | one pass, not measured | not measured |
| 1 | Claude Opus 5 | 100.0% 100.0% to 100.0% 40 records over 10 items, pooled from 4 entries | 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, rampstackco-skill-creation-walkthrough, yusufkaraaslan-skill-builder (of 4 entries) | $0.05962 run on subscription; shown at API list price for comparison | 313 in, 2,322 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 1 | Claude Sonnet 5 | 100.0% 100.0% to 100.0% 40 records over 10 items, pooled from 4 entries | 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, yusufkaraaslan-skill-builder (of 4 entries) | $0.02027 run on subscription; shown at API list price for comparison | 639 in, 1,899 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 2 | Claude Haiku 4.5 | 67.3% 59.7% to 74.9% 40 records over 10 items, pooled from 4 entries | 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, rampstackco-skill-creation-walkthrough, yusufkaraaslan-skill-builder (of 4 entries) | $0.00504 run on subscription; shown at API list price for comparison | 233 in, 961 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Spec writing · required sections and acceptance criteria
tested in Claude Code, Fable 5.1, single attempt
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 97.7% 96.2% to 99.2% 20 records over 10 items, pooled from 2 entries | 99.1% rampstackco-pm-spec-writing (of 2 entries) | $0.11240 run on subscription; shown at API list price for comparison | 388 in, 2,170 out | 85 | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Spec writing · required sections and acceptance criteria
tested in Claude Code, Wider run, three ways
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Haiku 4.5 | 99.2% 98.7% to 99.7% 60 records over 10 items, pooled from 6 entries | 99.1% github-prd (of 6 entries) | $0.00315 run on subscription; shown at API list price for comparison | 283 in, 574 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 2 | Claude Fable 5.1 | 97.9% 96.8% to 99.0% 60 records over 10 items, pooled from 6 entries | 100.0% rampstackco-pm-spec-writing (of 6 entries) | $0.10719 run on subscription; shown at API list price for comparison | 388 in, 2,066 out | 110 | one pass, not measured | not measured |
| 3 | Claude Sonnet 5 | 96.2% 94.9% to 97.6% 60 records over 10 items, pooled from 6 entries | 97.3% github-prd (of 6 entries) | $0.01601 run on subscription; shown at API list price for comparison | 712 in, 1,459 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
| 4 | Claude Opus 5 | 92.3% 91.1% to 93.5% 60 records over 10 items, pooled from 6 entries | 94.5% phuryn-create-prd (of 6 entries) | $0.06098 run on subscription; shown at API list price for comparison | 386 in, 2,362 out | 0 a gate holding at zero, not a measured zero | one pass, not measured | not measured |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Getting clean JSON back
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 0.0% 0.0% to 0.0% 50 records | 100.0% C01-json-schema | $0.04321 run on subscription; shown at API list price for comparison | 1,496 in, 565 out | 46 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 1 | Claude Fable 5.1 | 0.0% 0.0% to 0.0% 50 records | 100.0% C01-json-schema | $0.04114 run on subscription; shown at API list price for comparison | 1,506 in, 522 out | 0 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 1 | Claude Haiku 4.5 | 0.0% 0.0% to 0.0% 50 records | 100.0% C01-json-schema | $0.00316 run on subscription; shown at API list price for comparison | 1,102 in, 412 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 1 | Claude Opus 5 | 0.0% 0.0% to 0.0% 50 records | 100.0% C01-json-schema | $0.02287 run on subscription; shown at API list price for comparison | 1,496 in, 616 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 1 | Claude Sonnet 5 | 0.0% 0.0% to 0.0% 50 records | 100.0% C01-json-schema | $0.01160 run on subscription; shown at API list price for comparison | 3,126 in, 535 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Instructions before or after
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C02-instruction-after | $0.02645 run on subscription; shown at API list price for comparison | 2,465 in, 36 out | 0 | 20 of 20 every repeat agreed no sampling control here asked five times: withheld: 9 of 10 tasks got five usable answers | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 1 | Claude Fable 5.1 | 100.0% 100.0% to 100.0% 50 records | 100.0% C02-instruction-after | $0.02928 run on subscription; shown at API list price for comparison | 2,748 in, 36 out | 0 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 1 | Claude Haiku 4.5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C02-instruction-after | $0.00201 run on subscription; shown at API list price for comparison | 1,806 in, 41 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 1 | Claude Opus 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C02-instruction-after | $0.01323 run on subscription; shown at API list price for comparison | 2,465 in, 36 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 1 | Claude Sonnet 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C02-instruction-after | $0.00855 run on subscription; shown at API list price for comparison | 4,095 in, 36 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Using tags and formatting
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 100.0% 100.0% to 100.0% 48 records | 100.0% C03-xml-delimiters | $0.11688 run on subscription; shown at API list price for comparison | 2,370 in, 1,864 out | 543 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 1 | Claude Fable 5.1 | 100.0% 100.0% to 100.0% 50 records | 100.0% C03-xml-delimiters | $0.17308 run on subscription; shown at API list price for comparison | 2,193 in, 3,023 out | 1,351 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 1 | Claude Haiku 4.5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C03-xml-delimiters | $0.00533 run on subscription; shown at API list price for comparison | 1,574 in, 752 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 1 | Claude Opus 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C03-xml-delimiters | $0.04752 run on subscription; shown at API list price for comparison | 2,183 in, 1,464 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 1 | Claude Sonnet 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C03-xml-delimiters | $0.01651 run on subscription; shown at API list price for comparison | 3,813 in, 889 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Showing examples
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C04-three-shot-format | $0.03131 run on subscription; shown at API list price for comparison | 1,719 in, 283 out | 132 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 1 | Claude Fable 5.1 | 100.0% 100.0% to 100.0% 50 records | 100.0% C04-three-shot-format | $0.02474 run on subscription; shown at API list price for comparison | 1,729 in, 149 out | 0 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 1 | Claude Haiku 4.5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C04-three-shot-format | $0.00187 run on subscription; shown at API list price for comparison | 1,281 in, 118 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 1 | Claude Opus 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C04-three-shot-format | $0.01238 run on subscription; shown at API list price for comparison | 1,719 in, 151 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 1 | Claude Sonnet 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C04-three-shot-format | $0.00822 run on subscription; shown at API list price for comparison | 3,349 in, 152 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Showing examples
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 82.3% 78.6% to 86.0% 10 items | 99.7% C04b-zero-vs-one | $0.08170 run on subscription; shown at API list price for comparison | 2,621 in, 1,110 out | 872 | 18 of 20 7.1 percentage points apart where not no sampling control here asked five times: 8 of 10 the same every time, 49 to 94 percent, no sampling control here | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 2 | Claude Fable 5.1 | 81.7% 77.7% to 85.7% 10 items | 100.0% C04b-zero-vs-one | $0.05013 run on subscription; shown at API list price for comparison | 2,590 in, 485 out | 252 | 19 of 20 5.7 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 3 | Claude Haiku 4.5 | 81.4% 77.2% to 85.7% 10 items | 94.3% C04b-zero-vs-one | $0.00311 run on subscription; shown at API list price for comparison | 1,961 in, 230 out | 0 a gate holding at zero, not a measured zero | 18 of 20 5.7 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 3 | Claude Opus 5 | 81.4% 77.2% to 85.7% 10 items | 98.0% C04b-zero-vs-one | $0.01882 run on subscription; shown at API list price for comparison | 2,578 in, 237 out | 0 a gate holding at zero, not a measured zero | 18 of 20 7.1 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 3 | Claude Sonnet 5 | 81.4% 77.2% to 85.7% 10 items | 97.7% C04b-zero-vs-one | $0.01142 run on subscription; shown at API list price for comparison | 4,534 in, 235 out | 0 a gate holding at zero, not a measured zero | 18 of 20 7.1 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Showing examples
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 82.0% 78.2% to 85.8% 10 items | 100.0% C04b-zero-vs-three | $0.07734 run on subscription; shown at API list price for comparison | 2,578 in, 1,031 out | 798 | 19 of 20 8.6 percentage points apart where not no sampling control here asked five times: withheld: 9 of 10 tasks got five usable answers | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 2 | Claude Sonnet 5 | 81.7% 77.7% to 85.7% 10 items | 100.0% C04b-zero-vs-three | $0.01142 run on subscription; shown at API list price for comparison | 4,534 in, 236 out | 0 a gate holding at zero, not a measured zero | 19 of 20 5.7 percentage points apart where not no sampling control here asked five times: 9 of 10 the same every time, 60 to 98 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
| 3 | Claude Fable 5.1 | 81.4% 77.2% to 85.7% 10 items | 100.0% C04b-zero-vs-three | $0.05010 run on subscription; shown at API list price for comparison | 2,590 in, 484 out | 252 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 3 | Claude Haiku 4.5 | 81.4% 77.2% to 85.7% 10 items | 98.9% C04b-zero-vs-three | $0.00310 run on subscription; shown at API list price for comparison | 1,961 in, 228 out | 0 a gate holding at zero, not a measured zero | 19 of 20 5.7 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 3 | Claude Opus 5 | 81.4% 77.2% to 85.7% 10 items | 99.7% C04b-zero-vs-three | $0.01882 run on subscription; shown at API list price for comparison | 2,578 in, 237 out | 0 a gate holding at zero, not a measured zero | 19 of 20 5.7 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Better step-by-step answers
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 40.0% 26.3% to 53.7% 50 records | 50.0% C05-think-step-by-step | $0.02052 run on subscription; shown at API list price for comparison | 1,444 in, 122 out | 107 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 1 | Claude Fable 5.1 | 40.0% 26.3% to 53.7% 50 records | 50.0% C05-think-step-by-step | $0.02103 run on subscription; shown at API list price for comparison | 1,454 in, 130 out | 0 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 1 | Claude Opus 5 | 40.0% 26.3% to 53.7% 50 records | 46.0% C05-think-step-by-step | $0.00807 run on subscription; shown at API list price for comparison | 1,444 in, 34 out | 0 a gate holding at zero, not a measured zero | 19 of 20 60.0 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 1 | Claude Sonnet 5 | 40.0% 26.3% to 53.7% 50 records | 50.0% C05-think-step-by-step | $0.00630 run on subscription; shown at API list price for comparison | 3,074 in, 15 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
| 2 | Claude Haiku 4.5 | 20.0% 8.8% to 31.2% 50 records | 50.0% C05-think-step-by-step | $0.00135 run on subscription; shown at API list price for comparison | 1,098 in, 50 out | 0 a gate holding at zero, not a measured zero | 17 of 20 46.7 percentage points apart where not no sampling control here asked five times: 8 of 10 the same every time, 49 to 94 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Stop made-up answers
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 76.0% 64.0% to 88.0% 50 records | 100.0% C06-permit-idk | $0.03175 run on subscription; shown at API list price for comparison | 1,450 in, 345 out | 0 | 17 of 20 46.7 percentage points apart where not no sampling control here asked five times: 6 of 10 the same every time, 31 to 83 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 2 | Claude Sonnet 5 | 38.0% 24.4% to 51.6% 50 records | 100.0% C06-permit-idk | $0.01083 run on subscription; shown at API list price for comparison | 3,070 in, 469 out | 0 a gate holding at zero, not a measured zero | 16 of 20 50.0 percentage points apart where not no sampling control here asked five times: 5 of 10 the same every time, 24 to 76 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
| 3 | Claude Opus 5 | 30.0% 17.2% to 42.8% 50 records | 100.0% C06-permit-idk | $0.02363 run on subscription; shown at API list price for comparison | 1,440 in, 657 out | 0 a gate holding at zero, not a measured zero | 15 of 20 40.0 percentage points apart where not no sampling control here asked five times: 8 of 10 the same every time, 49 to 94 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 4 | Claude Fable 5 | 24.0% 12.0% to 36.0% 50 records | 100.0% C06-permit-idk | $0.03501 run on subscription; shown at API list price for comparison | 1,440 in, 412 out | 44 | 14 of 20 46.7 percentage points apart where not no sampling control here asked five times: withheld: 9 of 10 tasks got five usable answers | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 5 | Claude Haiku 4.5 | 22.0% 10.4% to 33.6% 50 records | 100.0% C06-permit-idk | $0.00340 run on subscription; shown at API list price for comparison | 1,081 in, 463 out | 0 a gate holding at zero, not a measured zero | 16 of 20 60.0 percentage points apart where not no sampling control here asked five times: 9 of 10 the same every time, 60 to 98 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Instructions in long prompts
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 100.0% 100.0% to 100.0% 33 records | 100.0% C07-instruction-at-end | $0.23998 run on subscription; shown at API list price for comparison | 4,267 in, 3,946 out | 783 | 17 of 17 every repeat agreed no sampling control here asked five times: withheld: 8 of 10 tasks got five usable answers | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 1 | Claude Fable 5.1 | 100.0% 100.0% to 100.0% 50 records | 100.0% C07-instruction-at-end | $0.16619 run on subscription; shown at API list price for comparison | 6,464 in, 2,031 out | 362 | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 1 | Claude Haiku 4.5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C07-instruction-at-end | $0.00664 run on subscription; shown at API list price for comparison | 2,211 in, 885 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 1 | Claude Opus 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C07-instruction-at-end | $0.08287 run on subscription; shown at API list price for comparison | 3,170 in, 2,681 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 1 | Claude Sonnet 5 | 100.0% 100.0% to 100.0% 50 records | 100.0% C07-instruction-at-end | $0.02143 run on subscription; shown at API list price for comparison | 4,800 in, 1,183 out | 0 a gate holding at zero, not a measured zero | 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Offering the model money
tested in Claude Code
| Rank | Model | Answer length, on its own | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5 | 544 words 509 to 579 50 answers | 828 words C08-tip-length | $0.15452 run on subscription; shown at API list price for comparison | 1,247 in, 5,931 out | 0 a gate holding at zero, not a measured zero | 0 of 20 112 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
| 2 | Claude Fable 5.1 | 403 words 385 to 421 50 answers | 508 words C08-tip-length | $0.23338 run on subscription; shown at API list price for comparison | 1,257 in, 4,416 out | 3 | 0 of 20 58 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 3 | Claude Sonnet 5 | 377 words 362 to 392 50 answers | 457 words C08-tip-length | $0.04774 run on subscription; shown at API list price for comparison | 2,877 in, 4,199 out | 0 a gate holding at zero, not a measured zero | 0 of 20 44 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
| 4 | Claude Fable 5 | 325 words 313 to 336 50 answers | 387 words C08-tip-length | $0.20286 run on subscription; shown at API list price for comparison | 1,247 in, 3,808 out | 86 | 0 of 20 32 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 5 | Claude Haiku 4.5 | 241 words 236 to 247 50 answers | 258 words C08-tip-length | $0.01031 run on subscription; shown at API list price for comparison | 928 in, 1,877 out | 0 a gate holding at zero, not a measured zero | 0 of 20 12 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Questions about recent events
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Haiku 4.5 | 96.7% 90.1% to 100.0%† 30 items | 99.2% C16-cutoff-disclosure | $0.00220 run on subscription; shown at API list price for comparison | 474 in, 345 out | 0 a gate holding at zero, not a measured zero | 19 of 20 50.0 percentage points apart where not 40 more answered once no sampling control here asked five times: 28 of 30 the same every time, 79 to 98 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 2 | Claude Sonnet 5 | 93.3% 84.3% to 100.0%† 30 items | 92.5% C16-cutoff-disclosure | $0.00757 run on subscription; shown at API list price for comparison | 1,448 in, 467 out | 0 a gate holding at zero, not a measured zero | 19 of 20 50.0 percentage points apart where not 40 more answered once no sampling control here asked five times: 22 of 30 the same every time, 56 to 86 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
| 3 | Claude Fable 5.1 | 92.2% 83.0% to 100.0%† 30 items | 89.2% C16-cutoff-disclosure | $0.04170 run on subscription; shown at API list price for comparison | 638 in, 706 out | 347 | 18 of 20 58.3 percentage points apart where not 40 more answered once no sampling control here asked five times: 20 of 30 the same every time, 49 to 81 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 4 | Claude Fable 5 | 87.8% 77.5% to 98.0% 30 items | 95.6% C16-cutoff-disclosure | $0.04206 run on subscription; shown at API list price for comparison | 633 in, 715 out | 336 | 15 of 20 60.0 percentage points apart where not 40 more answered once no sampling control here asked five times: 20 of 30 the same every time, 49 to 81 percent, no sampling control here | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 5 | Claude Opus 5 | 80.3% 67.4% to 93.1% 30 items | 86.7% C16-cutoff-disclosure | $0.01566 run on subscription; shown at API list price for comparison | 633 in, 500 out | 0 a gate holding at zero, not a measured zero | 13 of 20 59.5 percentage points apart where not 40 more answered once no sampling control here asked five times: 23 of 30 the same every time, 59 to 88 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Getting the length right
tested in Claude Code
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 76.0% 53.7% to 98.4% 10 items | 99.5% C17-exact-length | $0.16668 run on subscription; shown at API list price for comparison | 1,768 in, 2,980 out | 437 | 7 of 20 7.1 percentage points apart where not no sampling control here asked five times: 1 of 10 the same every time, 2 to 40 percent, no sampling control here | not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip |
| 2 | Claude Fable 5.1 | 50.2% 21.9% to 78.5% 10 items | 99.5% C17-exact-length | $0.22184 run on subscription; shown at API list price for comparison | 1,786 in, 4,080 out | 531 | 9 of 20 9.7 percentage points apart where not no sampling control here asked five times: 4 of 10 the same every time, 17 to 69 percent, no sampling control here | not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip |
| 3 | Claude Haiku 4.5 | 27.5% 0.0% to 54.9% 10 items | 92.5% C17-exact-length | $0.01432 run on subscription; shown at API list price for comparison | 1,319 in, 2,599 out | 0 a gate holding at zero, not a measured zero | 7 of 20 4.6 percentage points apart where not no sampling control here asked five times: 7 of 10 the same every time, 40 to 89 percent, no sampling control here | not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip |
| 4 | Claude Sonnet 5 | 17.8% 0.0% to 41.0%† 10 items | 95.2% C17-exact-length | $0.06459 run on subscription; shown at API list price for comparison | 4,050 in, 5,649 out | 0 a gate holding at zero, not a measured zero | 8 of 20 5.4 percentage points apart where not no sampling control here asked five times: 7 of 10 the same every time, 40 to 89 percent, no sampling control here | not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip |
| 5 | Claude Opus 5 | 15.0% 0.0% to 35.1%† 10 items | 97.3% C17-exact-length | $0.18907 run on subscription; shown at API list price for comparison | 1,768 in, 7,209 out | 0 a gate holding at zero, not a measured zero | 7 of 20 5.0 percentage points apart where not no sampling control here asked five times: 8 of 10 the same every time, 49 to 94 percent, no sampling control here | not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip |
This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.
† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
What changed between two versions
Higher on 5 jobs, lower on 1, no measurable change on 7, could not be measured on 7. 3 of these jobs were run in both skill tests, so each is counted twice.
Whether a figure moved because the version changed is not something this design can say. Neither version was run twice and thinking is not deterministic on this transport, so a moved figure and a re-run of the same version are not distinguishable here. Every difference below is reported as measured and never as explained.
Each version against the one before it, per job
Offering the model money
scores higher
Claude Fable 5 scored 324.7. Claude Fable 5.1 scored 403.2. The change is +78.5. It could sit between +61.3 and +95.7. Both versions did 10 tasks here.
word count, unbounded This scale has no fixed ceiling. It is built from the values here, so a position is a share of the largest figure on it.
Accessibility audit, wider run
no measurable change
Claude Fable 5 scored 0.897. Claude Fable 5.1 scored 0.879. The change is -0.018. It could sit between -0.099 and +0.063. Both versions did 10 tasks here.
Code review, wider run
no measurable change
Claude Fable 5 scored 0.858. Claude Fable 5.1 scored 0.890. The change is +0.032. It could sit between -0.025 and +0.089. Both versions did 10 tasks here.
On-page audit
scores higher
Claude Fable 5 scored 0.923. Claude Fable 5.1 scored 0.983. The change is +0.060. It could sit between +0.011 and +0.109. Both versions did 10 tasks here.
On-page audit, wider run
no measurable change
Claude Fable 5 scored 0.932. Claude Fable 5.1 scored 0.972. The change is +0.040. It could sit between -0.006 and +0.086. Both versions did 10 tasks here.
Skill authoring
could not be measured
Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.
Skill authoring, wider run
could not be measured
Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.
Spec writing
scores higher
Claude Fable 5 scored 0.932. Claude Fable 5.1 scored 0.977. The change is +0.045. It could sit between +0.022 and +0.068. Both versions did 10 tasks here.
Spec writing, wider run
scores higher
Claude Fable 5 scored 0.959. Claude Fable 5.1 scored 0.979. The change is +0.020. It could sit between +0.002 and +0.037. Both versions did 10 tasks here.
Getting clean JSON back
could not be measured
Claude Fable 5 scored 0.000. Claude Fable 5.1 scored 0.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.
Instructions before or after
could not be measured
Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.
Using tags and formatting
could not be measured
Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.
Showing examples
could not be measured
Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.
Showing examples
no measurable change
Claude Fable 5 scored 0.823. Claude Fable 5.1 scored 0.817. The change is -0.005. It could sit between -0.014 and +0.005. Both versions did 10 tasks here.
Showing examples
no measurable change
Claude Fable 5 scored 0.820. Claude Fable 5.1 scored 0.814. The change is -0.002. It could sit between -0.007 and +0.002. Both versions did 10 tasks here.
Better step-by-step answers
no measurable change
Claude Fable 5 scored 0.400. Claude Fable 5.1 scored 0.400. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.
Stop made-up answers
scores higher
Claude Fable 5 scored 0.240. Claude Fable 5.1 scored 0.760. The change is +0.520. It could sit between +0.324 and +0.716. Both versions did 10 tasks here.
Instructions in long prompts
could not be measured
Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 7 tasks here.
Questions about recent events
no measurable change
Claude Fable 5 scored 0.878. Claude Fable 5.1 scored 0.922. The change is +0.030. It could sit between -0.081 and +0.141. Both versions did 30 tasks here.
Getting the length right
scores lower
Claude Fable 5 scored 0.760. Claude Fable 5.1 scored 0.502. The change is -0.264. It could sit between -0.502 and -0.027. Both versions did 10 tasks here.
deterministic pass rate, 0 to 1 A bar wider than the scale is drawn to the edge with its cap left off. The table below gives its two ends.
Accessibility audit, wider run
On accessibility audit, wider run, Claude Fable 5.1 finds 88% of seeded accessibility defects unaided against 90% for Claude Fable 5; no measurable change.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 89.7% 80.3% to 99.1% 10 items | $0.04668 run on subscription; shown at API list price for comparison | 623 | tested in Claude Code, Fable 5 on the expansion design (three arms) |
| Claude Fable 5.1 | 87.9% 76.3% to 99.5% 10 items | $0.05126 run on subscription; shown at API list price for comparison | 755 | tested in Claude Code, Expansion cohort (three arms) |
Claude Fable 5.1 scored −1.8 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −9.9 and +6.3 points, which is why we call it no measurable change.
Code review, wider run
On code review, wider run, Claude Fable 5.1 finds 89% of seeded code defects unaided against 86% for Claude Fable 5; no measurable change.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 85.8% 77.4% to 94.2% 10 items | $0.04236 run on subscription; shown at API list price for comparison | 462 | tested in Claude Code, Fable 5 on the expansion design (three arms) |
| Claude Fable 5.1 | 89.0% 83.2% to 94.8% 10 items | $0.04548 run on subscription; shown at API list price for comparison | 555 | tested in Claude Code, Expansion cohort (three arms) |
Claude Fable 5.1 scored +3.2 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −2.5 and +8.9 points, which is why we call it no measurable change.
On-page audit
On on-page audit, Claude Fable 5.1 finds 98% of seeded on-page issues unaided against 92% for Claude Fable 5; scores higher.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 92.3% 85.3% to 99.4% 10 items | $0.03843 run on subscription; shown at API list price for comparison | 483 | tested in Claude Code, Skill cohort (one pass) |
| Claude Fable 5.1 | 98.3% 95.1% to 100.0%† 10 items | $0.03907 run on subscription; shown at API list price for comparison | 499 | tested in Claude Code, Fable 5.1, one pass |
Claude Fable 5.1 scored +6.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +1.1 and +10.9 points, which is why we call it scores higher.
On-page audit, wider run
On on-page audit, wider run, Claude Fable 5.1 finds 97% of seeded on-page issues unaided against 93% for Claude Fable 5; no measurable change.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 93.2% 86.4% to 100.0%† 10 items | $0.03795 run on subscription; shown at API list price for comparison | 431 | tested in Claude Code, Fable 5 on the expansion design (three arms) |
| Claude Fable 5.1 | 97.2% 93.5% to 100.0%† 10 items | $0.03873 run on subscription; shown at API list price for comparison | 491 | tested in Claude Code, Expansion cohort (three arms) |
Claude Fable 5.1 scored +4.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −0.6 and +8.6 points, which is why we call it no measurable change.
Skill authoring
On skill authoring, Claude Fable 5.1 finds 100% of specification requirements unaided against 100% for Claude Fable 5; could not be measured.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 100.0% 100.0% to 100.0% 10 items | $0.10060 run on subscription; shown at API list price for comparison | 64 | tested in Claude Code, Skill cohort (one pass) |
| Claude Fable 5.1 | 100.0% 100.0% to 100.0% 10 items | $0.15512 run on subscription; shown at API list price for comparison | 145 | tested in Claude Code, Fable 5.1, one pass |
Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.
Skill authoring, wider run
On skill authoring, wider run, Claude Fable 5.1 finds 100% of specification requirements unaided against 100% for Claude Fable 5; could not be measured.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 100.0% 100.0% to 100.0% 10 items | $0.09942 run on subscription; shown at API list price for comparison | 34 | tested in Claude Code, Fable 5 on the expansion design (three arms) |
| Claude Fable 5.1 | 100.0% 100.0% to 100.0% 10 items | $0.16037 run on subscription; shown at API list price for comparison | 137 | tested in Claude Code, Expansion cohort (three arms) |
Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.
Spec writing
On spec writing, Claude Fable 5.1 finds 98% of required sections and acceptance criteria unaided against 93% for Claude Fable 5; scores higher.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 93.2% 91.2% to 95.2% 10 items | $0.08369 run on subscription; shown at API list price for comparison | 20 | tested in Claude Code, Skill cohort (one pass) |
| Claude Fable 5.1 | 97.7% 96.2% to 99.2% 10 items | $0.11240 run on subscription; shown at API list price for comparison | 85 | tested in Claude Code, Fable 5.1, one pass |
Claude Fable 5.1 scored +4.5 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +2.2 and +6.8 points, which is why we call it scores higher.
Spec writing, wider run
On spec writing, wider run, Claude Fable 5.1 finds 98% of required sections and acceptance criteria unaided against 96% for Claude Fable 5; scores higher.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 95.9% 94.7% to 97.1% 10 items | $0.08887 run on subscription; shown at API list price for comparison | 13 | tested in Claude Code, Fable 5 on the expansion design (three arms) |
| Claude Fable 5.1 | 97.9% 96.8% to 99.0% 10 items | $0.10719 run on subscription; shown at API list price for comparison | 110 | tested in Claude Code, Expansion cohort (three arms) |
Claude Fable 5.1 scored +2.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.2 and +3.7 points, which is why we call it scores higher.
Getting clean JSON back
On getting clean JSON back, Claude Fable 5.1 scores 0% unaided against 0% for Claude Fable 5; could not be measured.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 0.0% 0.0% to 0.0% 50 records | $0.04321 run on subscription; shown at API list price for comparison | 46 | tested in Claude Code |
| Claude Fable 5.1 | 0.0% 0.0% to 0.0% 50 records | $0.04114 run on subscription; shown at API list price for comparison | 0 | tested in Claude Code |
Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.
Instructions before or after
On instructions before or after, Claude Fable 5.1 scores 100% unaided against 100% for Claude Fable 5; could not be measured.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 100.0% 100.0% to 100.0% 50 records | $0.02645 run on subscription; shown at API list price for comparison | 0 | tested in Claude Code |
| Claude Fable 5.1 | 100.0% 100.0% to 100.0% 50 records | $0.02928 run on subscription; shown at API list price for comparison | 0 | tested in Claude Code |
Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.
Using tags and formatting
On using tags and formatting, Claude Fable 5.1 scores 100% unaided against 100% for Claude Fable 5; could not be measured.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 100.0% 100.0% to 100.0% 48 records | $0.11688 run on subscription; shown at API list price for comparison | 543 | tested in Claude Code |
| Claude Fable 5.1 | 100.0% 100.0% to 100.0% 50 records | $0.17308 run on subscription; shown at API list price for comparison | 1,351 | tested in Claude Code |
Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.
Showing examples
On showing examples, Claude Fable 5.1 scores 100% unaided against 100% for Claude Fable 5; could not be measured.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 100.0% 100.0% to 100.0% 50 records | $0.03131 run on subscription; shown at API list price for comparison | 132 | tested in Claude Code |
| Claude Fable 5.1 | 100.0% 100.0% to 100.0% 50 records | $0.02474 run on subscription; shown at API list price for comparison | 0 | tested in Claude Code |
Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.
Showing examples
On showing examples, Claude Fable 5.1 scores 82% unaided against 82% for Claude Fable 5; no measurable change.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 82.3% 78.6% to 86.0% 10 items | $0.08170 run on subscription; shown at API list price for comparison | 872 | tested in Claude Code |
| Claude Fable 5.1 | 81.7% 77.7% to 85.7% 10 items | $0.05013 run on subscription; shown at API list price for comparison | 252 | tested in Claude Code |
Claude Fable 5.1 scored −0.5 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −1.4 and +0.5 points, which is why we call it no measurable change.
Showing examples
On showing examples, Claude Fable 5.1 scores 81% unaided against 82% for Claude Fable 5; no measurable change.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 82.0% 78.2% to 85.8% 10 items | $0.07734 run on subscription; shown at API list price for comparison | 798 | tested in Claude Code |
| Claude Fable 5.1 | 81.4% 77.2% to 85.7% 10 items | $0.05010 run on subscription; shown at API list price for comparison | 252 | tested in Claude Code |
Claude Fable 5.1 scored −0.2 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −0.7 and +0.2 points, which is why we call it no measurable change.
Better step-by-step answers
On better step-by-step answers, Claude Fable 5.1 scores 40% unaided against 40% for Claude Fable 5; no measurable change.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 40.0% 26.3% to 53.7% 50 records | $0.02052 run on subscription; shown at API list price for comparison | 107 | tested in Claude Code |
| Claude Fable 5.1 | 40.0% 26.3% to 53.7% 50 records | $0.02103 run on subscription; shown at API list price for comparison | 0 | tested in Claude Code |
Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it no measurable change.
Stop made-up answers
On stop made-up answers, Claude Fable 5.1 scores 76% unaided against 24% for Claude Fable 5; scores higher.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 24.0% 12.0% to 36.0% 50 records | $0.03501 run on subscription; shown at API list price for comparison | 44 | tested in Claude Code |
| Claude Fable 5.1 | 76.0% 64.0% to 88.0% 50 records | $0.03175 run on subscription; shown at API list price for comparison | 0 | tested in Claude Code |
Claude Fable 5.1 scored +52.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +32.4 and +71.6 points, which is why we call it scores higher.
Instructions in long prompts
On instructions in long prompts, Claude Fable 5.1 scores 100% unaided against 100% for Claude Fable 5; could not be measured.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 100.0% 100.0% to 100.0% 33 records | $0.23998 run on subscription; shown at API list price for comparison | 783 | tested in Claude Code |
| Claude Fable 5.1 | 100.0% 100.0% to 100.0% 50 records | $0.16619 run on subscription; shown at API list price for comparison | 362 | tested in Claude Code |
Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 7 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.
Offering the model money
On offering the model money, Claude Fable 5.1 scores 403 words unaided against 325 words for Claude Fable 5; scores higher.
| Version | Answer length, on its own | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 325 words 313 to 336 50 answers | $0.20286 run on subscription; shown at API list price for comparison | 86 | tested in Claude Code |
| Claude Fable 5.1 | 403 words 385 to 421 50 answers | $0.23338 run on subscription; shown at API list price for comparison | 3 | tested in Claude Code |
Claude Fable 5.1 scored +7852.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +6133.2 and +9570.8 points, which is why we call it scores higher.
Questions about recent events
On questions about recent events, Claude Fable 5.1 scores 92% unaided against 88% for Claude Fable 5; no measurable change.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 87.8% 77.5% to 98.0% 30 items | $0.04206 run on subscription; shown at API list price for comparison | 336 | tested in Claude Code |
| Claude Fable 5.1 | 92.2% 83.0% to 100.0%† 30 items | $0.04170 run on subscription; shown at API list price for comparison | 347 | tested in Claude Code |
Claude Fable 5.1 scored +3.0 points against Claude Fable 5, on the same 30 items both answered. Run it again and the gap would most likely land between −8.1 and +14.1 points, which is why we call it no measurable change.
Getting the length right
On getting the length right, Claude Fable 5.1 scores 50% unaided against 76% for Claude Fable 5; scores lower.
| Version | Unaided score | Cost per task | Thinking per task | Measured on |
|---|---|---|---|---|
| Claude Fable 5 | 76.0% 53.7% to 98.4% 10 items | $0.16668 run on subscription; shown at API list price for comparison | 437 | tested in Claude Code |
| Claude Fable 5.1 | 50.2% 21.9% to 78.5% 10 items | $0.22184 run on subscription; shown at API list price for comparison | 531 | tested in Claude Code |
Claude Fable 5.1 scored −26.4 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −50.2 and −2.7 points, which is why we call it scores lower.
This is the first point in a version series, not a rate. One step between two versions supports no curve and no guess at a third. Claude Fable 5 and Claude Fable 5.1 price identically, so a cost difference between them is a token difference.
Opus 5.5 beside Fable 5.1, and after Opus 5
Beside Fable 5.1
The maker says Opus 5.5 comes close to Fable 5.1. Both models think on every answer, so this is the fair pair. We measured one thing: whether the two likely ranges overlap on each job. Overlap means we cannot tell them apart here. It does not mean they are the same.
Prompting tips: the two likely ranges overlap on 14 of 14 jobs.
Every job, with its figures
| Job | Claude Opus 5.5 | Claude Fable 5.1 | Overlap | Cost per task |
|---|---|---|---|---|
| Getting clean JSON back | +100.0 [+100.0, +100.0] | +100.0 [+100.0, +100.0] | yes | $0.0045 / $0.0086 |
| Instructions before or after | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | yes | $0.0035 / $0.0053 |
| Using tags and formatting | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | yes | $0.0123 / $0.0355 |
| Showing examples | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | yes | $0.0033 / $0.0062 |
| Better step-by-step answers | +10.0 [-9.6, +29.6] | +10.0 [-9.6, +29.6] | yes | $0.0030 / $0.0058 |
| Stop made-up answers | +10.0 [-9.6, +29.6] | +30.0 [+0.1, +59.9] | yes | $0.0030 / $0.0049 |
| Instructions in long prompts | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | yes | $0.0115 / $0.0273 |
| Offering the model money | +125.5 words [+61.5 words, +189.5 words] | +58.8 words [+15.5 words, +102.1 words] | yes | $0.0295 / $0.0525 |
| Showing examples | +17.1 [+13.4, +20.9] | +17.1 [+13.4, +20.9] | yes | $0.0052 / $0.0086 |
| Showing examples | +17.1 [+13.4, +20.9] | +18.6 [+14.3, +22.8] | yes | $0.0049 / $0.0083 |
| Getting the length right | +19.9 [+2.2, +37.5] | +52.2 [+21.8, +82.5] | yes | $0.0215 / $0.0503 |
| Questions about recent events | -6.7 [-15.7, +2.4] | -10.0 [-20.9, +0.9] | yes | $0.0076 / $0.0165 |
| Getting clean JSON back, harder set | +100.0 [+100.0, +100.0] | +100.0 [+100.0, +100.0] | yes | $0.0071 / $0.0161 |
| Getting the length right, harder set | +99.8 [+99.3, +100.2] | +99.8 [+99.3, +100.2] | yes | $0.0157 / $0.0364 |
Cost is what each side's own answers would have cost at the maker's list price. Nobody was charged it: every answer ran on a subscription.
Tips over several turns: the two likely ranges overlap on 3 of 3 jobs.
Every job, with its figures
| Job | Claude Opus 5.5 | Claude Fable 5.1 | Overlap | Cost per task |
|---|---|---|---|---|
| Rules for the whole chat | +0.0 [+0.0, +0.0] | +4.0 [-1.2, +9.2] | yes | $0.2724 / $0.1998 |
| Splitting big tasks | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | yes | $0.0084 / $0.0132 |
| Prompt, then revise | +1.4 [-1.4, +4.2] | -1.4 [-4.2, +1.4] | yes | $0.0137 / $0.0325 |
Cost is what each side's own answers would have cost at the maker's list price. Nobody was charged it: every answer ran on a subscription.
Skills: the two likely ranges overlap on 22 of 25 jobs.
Every job, with its figures
| Job | Claude Opus 5.5 | Claude Fable 5.1 | Overlap | Cost per task |
|---|---|---|---|---|
S07-seo-onpage | +4.2 [-4.7, +13.0] | +5.4 [-1.7, +12.4] | yes | $0.0461 / $0.1273 |
S08-ecc-seo | -3.3 [-9.9, +3.2] | +6.2 [-7.4, +19.8] | yes | $0.0188 / $0.0585 |
S20-alirezarezvani-seo-audit | +5.0 [-4.8, +14.8] | +0.0 [+0.0, +0.0] | yes | $0.0465 / $0.1269 |
S03-accessibility-audit | +0.0 [+0.0, +0.0] | +4.0 [-1.2, +9.2] | yes | $0.0881 / $0.2569 |
S04-ecc-accessibility | +2.1 [-5.1, +9.3] | -6.0 [-14.4, +2.4] | yes | $0.0266 / $0.1000 |
S14-wshobson-wcag-audit-patterns | -2.8 [-11.2, +5.7] | -2.0 [-5.9, +1.9] | yes | $0.0391 / $0.1331 |
S15-alirezarezvani-a11y-audit | -0.7 [-5.6, +4.1] | +2.0 [-7.1, +11.1] | yes | $0.1153 / $0.3322 |
S01-skill-creation-walkthrough | +0.0 [+0.0, +0.0] | -14.5 [-33.6, +4.5] | yes | $0.1230 / $0.3251 |
S02-skill-creator | +0.0 [+0.0, +0.0] | -43.6 [-66.9, -20.4] | no | $0.1287 / $0.4005 |
S12-openclaw-skill-creator | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | yes | $0.0470 / $0.1088 |
S13-yusufkaraaslan-skill-builder | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | yes | $0.0749 / $0.1633 |
S05-pm-spec-writing | +0.0 [+0.0, +0.0] | +3.6 [+0.7, +6.5] | no | $0.0912 / $0.2096 |
S06-product-capability | +1.8 [-0.6, +4.2] | +0.9 [-2.3, +4.1] | yes | $0.0553 / $0.1280 |
S16-phuryn-create-prd | +0.9 [-0.9, +2.7] | +3.6 [+0.7, +6.5] | yes | $0.0529 / $0.1146 |
S17-alirezarezvani-spec-driven-workflow | -0.9 [-4.1, +2.3] | -13.6 [-16.6, -10.7] | no | $0.1463 / $0.4132 |
S18-addyosmani-spec-driven-development | +0.9 [-2.3, +4.1] | +0.0 [-2.7, +2.7] | yes | $0.0649 / $0.1590 |
S19-github-prd | +1.8 [-0.6, +4.2] | -0.9 [-5.1, +3.2] | yes | $0.0584 / $0.1462 |
S11-code-review-web | +2.0 [-1.9, +5.9] | +0.0 [+0.0, +0.0] | yes | $0.0606 / $0.1734 |
S22-mattpocock-code-review | -3.7 [-8.5, +1.1] | +1.7 [-1.6, +4.9] | yes | $0.0233 / $0.0764 |
S23-alirezarezvani-pr-review-expert | +1.7 [-1.6, +4.9] | -1.7 [-4.9, +1.6] | yes | $0.0361 / $0.1123 |
S24-addyosmani-code-review-and-quality | -3.3 [-7.7, +1.0] | +0.0 [+0.0, +0.0] | yes | $0.0412 / $0.1235 |
S25-jeffallan-code-reviewer | -1.7 [-4.9, +1.6] | +0.0 [+0.0, +0.0] | yes | $0.0634 / $0.1846 |
S26-everyinc-ce-code-review | +3.3 [-3.2, +9.9] | +0.8 [-5.4, +7.0] | yes | $0.4647 / $1.1956 |
S27-wshobson-code-review-excellence | +0.0 [+0.0, +0.0] | -5.7 [-11.3, +0.02] | yes | $0.0343 / $0.1052 |
S28-open-gsd-code-review | +3.3 [-3.2, +9.9] | +0.0 [+0.0, +0.0] | yes | $0.0165 / $0.0642 |
Cost is what each side's own answers would have cost at the maker's list price. Nobody was charged it: every answer ran on a subscription.
Thinking was recorded rather than refused, by prior registration
Alone among the models tested this way, this one was set up in advance to have its thinking COUNTED rather than turned off, so thinking on it is not a failure. We wrote that down before the run precisely so that a result on this model could not later be explained away, or quietly discarded, on the grounds that it thought. The figures beside this are a measurement; the zeroes on the other models are a switch holding, and the two are not the same kind of number.
- Answers reporting a nonzero count
- 492 of 1,330
- Thinking tokens recorded
- 143,447
This model was also run through Claude Code, on a path that bills nothing. How it was called, and what that means for every cost figure on this page, is below.
Show the full breakdown, tested in Claude Code
How this model was called, tested in Claude Code
On the Claude Code CLI, on a subscription path with no API key and nothing billed. Every cost figure this path returns is a client-side estimate at list price and is recorded under that label and no other. What that means, and how it was calibrated.
These results are tested in Claude Code and are never averaged with any figure tested via API. Where this model was measured on both, the two are reported as two panels above.