Best AI Model by Task, With Cost per Task

Models are compared only on identical work. The same job, run the same way, on the same occasion. There is no table on this page that mixes two of any of those, and no figure anywhere that was averaged across them.

That is why there are many small tables rather than one large one. A single ranking across the jobs would be a number that describes no measurement. The jobs have different answer keys, different numbers of items and different scorers.

Every figure here is worked out from the committed answers. Nothing on this page was written by hand, and the methodology sets out how each was measured.

These tables do not rank models on how often they repeat themselves. A model held at temperature 0 repeats itself by construction. A model at its vendor's default does not, and one tested in Claude Code has no such setting at all. The setting sits beside each figure, and comparing across settings would rank the settings.

Cost against score, one job at a time

Pick a job. Each mark is one model: how much one task cost it with no help, against the score it got. The tables below carry every figure.

What one task costs with no help, and what it scores Accessibility audit

  • tested via API
  • tested in Claude Code

Across: what one task costs with no help, in US dollars at list price. Each gridline is ten times the one before it. Up: deterministic pass rate, 0 to 1.

  1. Gemini 3.1 Flash Lite First run, with and without the skill tested via API $0.00022 per task scored 63.3%, 46.0% to 80.7%
  2. Gemini 3.1 Flash Lite Wider run, three ways tested via API $0.00022 per task scored 63.3%, 46.0% to 80.7%
  3. GPT-5 mini First run, with and without the skill tested via API $0.00035 per task scored 69.9%, 54.2% to 85.5%
  4. GPT-5 mini Wider run, three ways tested via API $0.00036 per task scored 58.2%, 47.9% to 68.4%
  5. Claude Haiku 4.5 First run, with and without the skill tested via API $0.00088 per task scored 40.0%, 26.3% to 53.7%
  6. Claude Haiku 4.5 Wider run, three ways tested in Claude Code $0.00104 per task scored 46.1%, 34.1% to 58.1%
  7. Claude Haiku 4.5 Second run, every task twice tested in Claude Code $0.00206 per task scored 45.5%, 35.0% to 56.0%
  8. Claude Sonnet 5 Wider run, three ways tested in Claude Code $0.00368 per task scored 80.2%, 71.4% to 88.9%
  9. Claude Opus 5 First run, single attempt tested in Claude Code $0.00696 per task scored 89.0%, 79.7% to 98.3%
  10. Claude Opus 5 Wider run, three ways tested in Claude Code $0.00704 per task scored 87.1%, 78.0% to 96.2%
  11. Claude Sonnet 5 Second run, every task twice tested in Claude Code $0.00745 per task scored 82.3%, 73.9% to 90.7%
  12. Claude Opus 5 Second run, every task twice tested in Claude Code $0.01384 per task scored 87.3%, 78.3% to 96.4%
  13. Claude Opus 5.5 Opus 5.5 series, wider run, three ways tested in Claude Code $0.01591 per task scored 85.8%, 74.4% to 97.2% 18.3 s per call, middle half 16.7 s to 19.6 s
  14. Claude Fable 5 Fable 5, wider run, three ways tested in Claude Code $0.04668 per task scored 89.7%, 80.3% to 99.1% 22.5 s per call, middle half 19.0 s to 27.1 s
  15. Claude Fable 5.1 Fable 5.1, single attempt tested in Claude Code $0.05119 per task scored 85.8%, 72.6% to 99.1%
  16. Claude Fable 5.1 Wider run, three ways tested in Claude Code $0.05126 per task scored 87.9%, 76.3% to 99.5%
  17. Marks here are all one size. Not every mark has a time behind it. Sizing only the ones that do would draw the rest at a size we did not measure. Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Each mark is one model on one job: what a task cost it with no help, against the score it got. The bar through a mark shows how much that score could move if we ran it again. Shape and colour say which test method the model was reached through. The two test methods are measured on their own and are never ranked against each other. Compare marks of one colour with each other; a mark of one colour sitting above a mark of another is not a result. The tables are the source, and every figure drawn here is printed above. A model measured more than once appears once per test, named by test. Two tests are two measurements and are never averaged into one mark. Two marks here sit at the same cost and the same score, so they are drawn on top of each other and read as one. The list above has both. Names sit beside their marks where there is room for them, moved down where two would have overlapped, and on a narrow screen they are in the list above instead. The list pairs every name with its figures at every width.

What changed between two versions

Higher on 5 jobs, lower on 1, no measurable change on 7, could not be measured on 7. 3 of these jobs were run in both skill tests, so each is counted twice.

Whether a figure moved because the version changed is not something this design can say. Neither version was run twice and thinking is not deterministic on this transport, so a moved figure and a re-run of the same version are not distinguishable here. Every difference below is reported as measured and never as explained.

Each version against the one before it, per job

Offering the model money

scores higher

Claude Fable 5 scored 324.7. Claude Fable 5.1 scored 403.2. The change is +78.5. It could sit between +61.3 and +95.7. Both versions did 10 tasks here.

word count, unbounded This scale has no fixed ceiling. It is built from the values here, so a position is a share of the largest figure on it.

Accessibility audit, wider run

no measurable change

Claude Fable 5 scored 0.897. Claude Fable 5.1 scored 0.879. The change is -0.018. It could sit between -0.099 and +0.063. Both versions did 10 tasks here.

Code review, wider run

no measurable change

Claude Fable 5 scored 0.858. Claude Fable 5.1 scored 0.890. The change is +0.032. It could sit between -0.025 and +0.089. Both versions did 10 tasks here.

On-page audit

scores higher

Claude Fable 5 scored 0.923. Claude Fable 5.1 scored 0.983. The change is +0.060. It could sit between +0.011 and +0.109. Both versions did 10 tasks here.

On-page audit, wider run

no measurable change

Claude Fable 5 scored 0.932. Claude Fable 5.1 scored 0.972. The change is +0.040. It could sit between -0.006 and +0.086. Both versions did 10 tasks here.

Skill authoring

could not be measured

Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.

Skill authoring, wider run

could not be measured

Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.

Spec writing

scores higher

Claude Fable 5 scored 0.932. Claude Fable 5.1 scored 0.977. The change is +0.045. It could sit between +0.022 and +0.068. Both versions did 10 tasks here.

Spec writing, wider run

scores higher

Claude Fable 5 scored 0.959. Claude Fable 5.1 scored 0.979. The change is +0.020. It could sit between +0.002 and +0.037. Both versions did 10 tasks here.

Getting clean JSON back

could not be measured

Claude Fable 5 scored 0.000. Claude Fable 5.1 scored 0.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.

Instructions before or after

could not be measured

Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.

Using tags and formatting

could not be measured

Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.

Showing examples

could not be measured

Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.

Showing examples

no measurable change

Claude Fable 5 scored 0.823. Claude Fable 5.1 scored 0.817. The change is -0.005. It could sit between -0.014 and +0.005. Both versions did 10 tasks here.

Showing examples

no measurable change

Claude Fable 5 scored 0.820. Claude Fable 5.1 scored 0.814. The change is -0.002. It could sit between -0.007 and +0.002. Both versions did 10 tasks here.

Better step-by-step answers

no measurable change

Claude Fable 5 scored 0.400. Claude Fable 5.1 scored 0.400. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 10 tasks here.

Stop made-up answers

scores higher

Claude Fable 5 scored 0.240. Claude Fable 5.1 scored 0.760. The change is +0.520. It could sit between +0.324 and +0.716. Both versions did 10 tasks here.

Instructions in long prompts

could not be measured

Claude Fable 5 scored 1.000. Claude Fable 5.1 scored 1.000. The change is +0.000. It could sit between +0.000 and +0.000. Both versions did 7 tasks here.

Questions about recent events

no measurable change

Claude Fable 5 scored 0.878. Claude Fable 5.1 scored 0.922. The change is +0.030. It could sit between -0.081 and +0.141. Both versions did 30 tasks here.

Getting the length right

scores lower

Claude Fable 5 scored 0.760. Claude Fable 5.1 scored 0.502. The change is -0.264. It could sit between -0.502 and -0.027. Both versions did 10 tasks here.

deterministic pass rate, 0 to 1 A bar wider than the scale is drawn to the edge with its cap left off. The table below gives its two ends.

The hollow mark is the older version. The filled mark is the newer one. The bar shows how much the change could move if we ran it again. It starts at the older mark, so it covers where the newer mark could have landed. No colour here says which way a figure went. These words describe the number. They do not say one version is better. Work measured in different units gets its own scale. The tables below carry every figure. On some rows the older figure plus the change does not come out at the newer figure. Both are right. A mark is the average over every run of a task. The change is the average of the per-task differences. The two agree only when every task was run the same number of times.

Accessibility audit, wider run

On accessibility audit, wider run, Claude Fable 5.1 finds 88% of seeded accessibility defects unaided against 90% for Claude Fable 5; no measurable change.

Accessibility audit, wider run: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 89.7% 80.3% to 99.1% 10 items $0.04668 run on subscription; shown at API list price for comparison 623tested in Claude Code, Fable 5 on the expansion design (three arms)
Claude Fable 5.1 87.9% 76.3% to 99.5% 10 items $0.05126 run on subscription; shown at API list price for comparison 755tested in Claude Code, Expansion cohort (three arms)

Claude Fable 5.1 scored −1.8 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −9.9 and +6.3 points, which is why we call it no measurable change.

Code review, wider run

On code review, wider run, Claude Fable 5.1 finds 89% of seeded code defects unaided against 86% for Claude Fable 5; no measurable change.

Code review, wider run: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 85.8% 77.4% to 94.2% 10 items $0.04236 run on subscription; shown at API list price for comparison 462tested in Claude Code, Fable 5 on the expansion design (three arms)
Claude Fable 5.1 89.0% 83.2% to 94.8% 10 items $0.04548 run on subscription; shown at API list price for comparison 555tested in Claude Code, Expansion cohort (three arms)

Claude Fable 5.1 scored +3.2 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −2.5 and +8.9 points, which is why we call it no measurable change.

On-page audit

On on-page audit, Claude Fable 5.1 finds 98% of seeded on-page issues unaided against 92% for Claude Fable 5; scores higher.

On-page audit: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 92.3% 85.3% to 99.4% 10 items $0.03843 run on subscription; shown at API list price for comparison 483tested in Claude Code, Skill cohort (one pass)
Claude Fable 5.1 98.3% 95.1% to 100.0%† 10 items $0.03907 run on subscription; shown at API list price for comparison 499tested in Claude Code, Fable 5.1, one pass

Claude Fable 5.1 scored +6.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +1.1 and +10.9 points, which is why we call it scores higher.

On-page audit, wider run

On on-page audit, wider run, Claude Fable 5.1 finds 97% of seeded on-page issues unaided against 93% for Claude Fable 5; no measurable change.

On-page audit, wider run: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 93.2% 86.4% to 100.0%† 10 items $0.03795 run on subscription; shown at API list price for comparison 431tested in Claude Code, Fable 5 on the expansion design (three arms)
Claude Fable 5.1 97.2% 93.5% to 100.0%† 10 items $0.03873 run on subscription; shown at API list price for comparison 491tested in Claude Code, Expansion cohort (three arms)

Claude Fable 5.1 scored +4.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −0.6 and +8.6 points, which is why we call it no measurable change.

Skill authoring

On skill authoring, Claude Fable 5.1 finds 100% of specification requirements unaided against 100% for Claude Fable 5; could not be measured.

Skill authoring: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 100.0% 100.0% to 100.0% 10 items $0.10060 run on subscription; shown at API list price for comparison 64tested in Claude Code, Skill cohort (one pass)
Claude Fable 5.1 100.0% 100.0% to 100.0% 10 items $0.15512 run on subscription; shown at API list price for comparison 145tested in Claude Code, Fable 5.1, one pass

Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.

Skill authoring, wider run

On skill authoring, wider run, Claude Fable 5.1 finds 100% of specification requirements unaided against 100% for Claude Fable 5; could not be measured.

Skill authoring, wider run: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 100.0% 100.0% to 100.0% 10 items $0.09942 run on subscription; shown at API list price for comparison 34tested in Claude Code, Fable 5 on the expansion design (three arms)
Claude Fable 5.1 100.0% 100.0% to 100.0% 10 items $0.16037 run on subscription; shown at API list price for comparison 137tested in Claude Code, Expansion cohort (three arms)

Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.

Spec writing

On spec writing, Claude Fable 5.1 finds 98% of required sections and acceptance criteria unaided against 93% for Claude Fable 5; scores higher.

Spec writing: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 93.2% 91.2% to 95.2% 10 items $0.08369 run on subscription; shown at API list price for comparison 20tested in Claude Code, Skill cohort (one pass)
Claude Fable 5.1 97.7% 96.2% to 99.2% 10 items $0.11240 run on subscription; shown at API list price for comparison 85tested in Claude Code, Fable 5.1, one pass

Claude Fable 5.1 scored +4.5 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +2.2 and +6.8 points, which is why we call it scores higher.

Spec writing, wider run

On spec writing, wider run, Claude Fable 5.1 finds 98% of required sections and acceptance criteria unaided against 96% for Claude Fable 5; scores higher.

Spec writing, wider run: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 95.9% 94.7% to 97.1% 10 items $0.08887 run on subscription; shown at API list price for comparison 13tested in Claude Code, Fable 5 on the expansion design (three arms)
Claude Fable 5.1 97.9% 96.8% to 99.0% 10 items $0.10719 run on subscription; shown at API list price for comparison 110tested in Claude Code, Expansion cohort (three arms)

Claude Fable 5.1 scored +2.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.2 and +3.7 points, which is why we call it scores higher.

Getting clean JSON back

On getting clean JSON back, Claude Fable 5.1 scores 0% unaided against 0% for Claude Fable 5; could not be measured.

Getting clean JSON back: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 0.0% 0.0% to 0.0% 50 records $0.04321 run on subscription; shown at API list price for comparison 46tested in Claude Code
Claude Fable 5.1 0.0% 0.0% to 0.0% 50 records $0.04114 run on subscription; shown at API list price for comparison 0tested in Claude Code

Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.

Instructions before or after

On instructions before or after, Claude Fable 5.1 scores 100% unaided against 100% for Claude Fable 5; could not be measured.

Instructions before or after: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 100.0% 100.0% to 100.0% 50 records $0.02645 run on subscription; shown at API list price for comparison 0tested in Claude Code
Claude Fable 5.1 100.0% 100.0% to 100.0% 50 records $0.02928 run on subscription; shown at API list price for comparison 0tested in Claude Code

Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.

Using tags and formatting

On using tags and formatting, Claude Fable 5.1 scores 100% unaided against 100% for Claude Fable 5; could not be measured.

Using tags and formatting: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 100.0% 100.0% to 100.0% 48 records $0.11688 run on subscription; shown at API list price for comparison 543tested in Claude Code
Claude Fable 5.1 100.0% 100.0% to 100.0% 50 records $0.17308 run on subscription; shown at API list price for comparison 1,351tested in Claude Code

Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.

Showing examples

On showing examples, Claude Fable 5.1 scores 100% unaided against 100% for Claude Fable 5; could not be measured.

Showing examples: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 100.0% 100.0% to 100.0% 50 records $0.03131 run on subscription; shown at API list price for comparison 132tested in Claude Code
Claude Fable 5.1 100.0% 100.0% to 100.0% 50 records $0.02474 run on subscription; shown at API list price for comparison 0tested in Claude Code

Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.

Showing examples

On showing examples, Claude Fable 5.1 scores 82% unaided against 82% for Claude Fable 5; no measurable change.

Showing examples: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 82.3% 78.6% to 86.0% 10 items $0.08170 run on subscription; shown at API list price for comparison 872tested in Claude Code
Claude Fable 5.1 81.7% 77.7% to 85.7% 10 items $0.05013 run on subscription; shown at API list price for comparison 252tested in Claude Code

Claude Fable 5.1 scored −0.5 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −1.4 and +0.5 points, which is why we call it no measurable change.

Showing examples

On showing examples, Claude Fable 5.1 scores 81% unaided against 82% for Claude Fable 5; no measurable change.

Showing examples: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 82.0% 78.2% to 85.8% 10 items $0.07734 run on subscription; shown at API list price for comparison 798tested in Claude Code
Claude Fable 5.1 81.4% 77.2% to 85.7% 10 items $0.05010 run on subscription; shown at API list price for comparison 252tested in Claude Code

Claude Fable 5.1 scored −0.2 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −0.7 and +0.2 points, which is why we call it no measurable change.

Better step-by-step answers

On better step-by-step answers, Claude Fable 5.1 scores 40% unaided against 40% for Claude Fable 5; no measurable change.

Better step-by-step answers: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 40.0% 26.3% to 53.7% 50 records $0.02052 run on subscription; shown at API list price for comparison 107tested in Claude Code
Claude Fable 5.1 40.0% 26.3% to 53.7% 50 records $0.02103 run on subscription; shown at API list price for comparison 0tested in Claude Code

Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it no measurable change.

Stop made-up answers

On stop made-up answers, Claude Fable 5.1 scores 76% unaided against 24% for Claude Fable 5; scores higher.

Stop made-up answers: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 24.0% 12.0% to 36.0% 50 records $0.03501 run on subscription; shown at API list price for comparison 44tested in Claude Code
Claude Fable 5.1 76.0% 64.0% to 88.0% 50 records $0.03175 run on subscription; shown at API list price for comparison 0tested in Claude Code

Claude Fable 5.1 scored +52.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +32.4 and +71.6 points, which is why we call it scores higher.

Instructions in long prompts

On instructions in long prompts, Claude Fable 5.1 scores 100% unaided against 100% for Claude Fable 5; could not be measured.

Instructions in long prompts: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 100.0% 100.0% to 100.0% 33 records $0.23998 run on subscription; shown at API list price for comparison 783tested in Claude Code
Claude Fable 5.1 100.0% 100.0% to 100.0% 50 records $0.16619 run on subscription; shown at API list price for comparison 362tested in Claude Code

Claude Fable 5.1 scored +0.0 points against Claude Fable 5, on the same 7 items both answered. Run it again and the gap would most likely land between +0.0 and +0.0 points, which is why we call it could not be measured.

Offering the model money

On offering the model money, Claude Fable 5.1 scores 403 words unaided against 325 words for Claude Fable 5; scores higher.

Offering the model money: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionAnswer length, on its ownCost per taskThinking per taskMeasured on
Claude Fable 5 325 words 313 to 336 50 answers $0.20286 run on subscription; shown at API list price for comparison 86tested in Claude Code
Claude Fable 5.1 403 words 385 to 421 50 answers $0.23338 run on subscription; shown at API list price for comparison 3tested in Claude Code

Claude Fable 5.1 scored +7852.0 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between +6133.2 and +9570.8 points, which is why we call it scores higher.

Questions about recent events

On questions about recent events, Claude Fable 5.1 scores 92% unaided against 88% for Claude Fable 5; no measurable change.

Questions about recent events: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 87.8% 77.5% to 98.0% 30 items $0.04206 run on subscription; shown at API list price for comparison 336tested in Claude Code
Claude Fable 5.1 92.2% 83.0% to 100.0%† 30 items $0.04170 run on subscription; shown at API list price for comparison 347tested in Claude Code

Claude Fable 5.1 scored +3.0 points against Claude Fable 5, on the same 30 items both answered. Run it again and the gap would most likely land between −8.1 and +14.1 points, which is why we call it no measurable change.

Getting the length right

On getting the length right, Claude Fable 5.1 scores 50% unaided against 76% for Claude Fable 5; scores lower.

Getting the length right: Claude Fable 5 against Claude Fable 5.1, unaided.
VersionUnaided scoreCost per taskThinking per taskMeasured on
Claude Fable 5 76.0% 53.7% to 98.4% 10 items $0.16668 run on subscription; shown at API list price for comparison 437tested in Claude Code
Claude Fable 5.1 50.2% 21.9% to 78.5% 10 items $0.22184 run on subscription; shown at API list price for comparison 531tested in Claude Code

Claude Fable 5.1 scored −26.4 points against Claude Fable 5, on the same 10 items both answered. Run it again and the gap would most likely land between −50.2 and −2.7 points, which is why we call it scores lower.

This is the first point in a version series, not a rate. One step between two versions supports no curve and no guess at a third. Claude Fable 5 and Claude Fable 5.1 price identically, so a cost difference between them is a token difference.

Skill classes

Accessibility audit · seeded accessibility defects

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, First run, with and without the skill

Accessibility audit, tested via API, First run, with and without the skill. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5 mini 69.9% 54.2% to 85.5% 20 records over 10 items, pooled from 2 entries 68.9% rampstackco-accessibility-audit (of 2 entries) $0.00035 billed through the API 515 in, 111 out not measured one pass, not measured not measured
2Gemini 3.1 Flash Lite 63.3% 46.0% to 80.7% 20 records over 10 items, pooled from 2 entries 72.8% rampstackco-accessibility-audit (of 2 entries) $0.00022 billed through the API 528 in, 59 out not measured one pass, not measured not measured
3Claude Haiku 4.5 40.0% 26.3% to 53.7% 20 records over 10 items, pooled from 2 entries 69.9% rampstackco-accessibility-audit (of 2 entries) $0.00088 billed through the API 600 in, 55 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 20 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Accessibility audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 63.3% interval overlaps the best $0.00022 billed through the API $0.22
GPT-5 mini 69.9% the best score here $0.00035 billed through the API $0.35
Claude Haiku 4.5 40.0% interval does not reach the best $0.00088 billed through the API $0.88

Good enough pick. Gemini 3.1 Flash Lite at 63.3% and $0.22 for 1,000 tasks a month is the cheapest model whose bracket overlaps GPT-5 mini, the best on this table at 69.9% and $0.35.

tested via API, Wider run, three ways

Accessibility audit, tested via API, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Gemini 3.1 Flash Lite 63.3% 46.0% to 80.7% 40 records over 10 items, pooled from 4 entries 76.3% alirezarezvani-a11y-audit (of 4 entries) $0.00022 billed through the API 528 in, 59 out not measured one pass, not measured not measured
2GPT-5 mini 58.2% 47.9% to 68.4% 40 records over 10 items, pooled from 4 entries 62.6% rampstackco-accessibility-audit (of 4 entries) $0.00035 billed through the API 515 in, 115 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 40 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Accessibility audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 63.3% the best score here $0.00022 billed through the API $0.22
GPT-5 mini 58.2% interval overlaps the best $0.00035 billed through the API $0.35

Good enough pick. Gemini 3.1 Flash Lite at 63.3% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $0.22 for 1,000 tasks a month.

tested in Claude Code, First run, single attempt

Accessibility audit, tested in Claude Code, First run, single attempt. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5 89.0% 79.7% to 98.3% 20 records over 10 items, pooled from 2 entries 87.0% rampstackco-accessibility-audit (of 2 entries) $0.00696 run on subscription; shown at API list price for comparison 946 in, 89 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Accessibility audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5 89.0% the best score here $0.00696 run on subscription; shown at API list price for comparison $6.96

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Second run, every task twice

Accessibility audit, tested in Claude Code, Second run, every task twice. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5 87.3% 78.3% to 96.4% 40 records over 10 items, pooled from 2 entries 88.0% affaan-m-accessibility (of 2 entries) $0.01384 run on subscription; shown at API list price for comparison 1,892 in, 175 out 0 a gate holding at zero, not a measured zero 38 of 40 20.0 percentage points apart where not no sampling control here not measured
2Claude Sonnet 5 82.3% 73.9% to 90.7% 40 records over 10 items, pooled from 2 entries 82.3% affaan-m-accessibility (of 2 entries) $0.00745 run on subscription; shown at API list price for comparison 2,544 in, 237 out 0 a gate holding at zero, not a measured zero 31 of 40 22.8 percentage points apart where not no sampling control here not measured
3Claude Haiku 4.5 45.5% 35.0% to 56.0% 40 records over 10 items, pooled from 2 entries 69.0% rampstackco-accessibility-audit (of 2 entries) $0.00206 run on subscription; shown at API list price for comparison 1,530 in, 106 out 0 a gate holding at zero, not a measured zero 26 of 40 22.9 percentage points apart where not no sampling control here not measured
  • Claude Fable 5: not published. 71 of 76 of the no-help calls in this job were answered by a different model (Claude Opus 5). We hold the figure back until those calls are bought again and answered by the model we asked for. The calls that went astray are not a random few. They land on the very runs this column averages, so a score built from the ones that came back correctly would describe which model the platform chose to send, not what this model can do. Why some figures are held back, under the rule that governs it.

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Accessibility audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 45.5% interval does not reach the best $0.00206 run on subscription; shown at API list price for comparison $2.06
Claude Sonnet 5 82.3% interval overlaps the best $0.00745 run on subscription; shown at API list price for comparison $7.45
Claude Opus 5 87.3% the best score here $0.01384 run on subscription; shown at API list price for comparison $13.84

Good enough pick. Claude Sonnet 5 at 82.3% and $7.45 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Opus 5, the best on this table at 87.3% and $13.84.

tested in Claude Code, Fable 5.1, single attempt

Accessibility audit, tested in Claude Code, Fable 5.1, single attempt. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5.1 85.8% 72.6% to 99.1% 20 records over 10 items, pooled from 2 entries 93.5% rampstackco-accessibility-audit (of 2 entries) $0.05119 run on subscription; shown at API list price for comparison 948 in, 834 out 755 one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Accessibility audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5.1 85.8% the best score here $0.05119 run on subscription; shown at API list price for comparison $51.19

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Wider run, three ways

Accessibility audit, tested in Claude Code, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5.1 87.9% 76.3% to 99.5% 40 records over 10 items, pooled from 4 entries 91.5% alirezarezvani-a11y-audit (of 4 entries) $0.05126 run on subscription; shown at API list price for comparison 949 in, 835 out 755 one pass, not measured not measured
2Claude Opus 5 87.1% 78.0% to 96.2% 40 records over 10 items, pooled from 4 entries 91.5% affaan-m-accessibility (of 4 entries) $0.00704 run on subscription; shown at API list price for comparison 946 in, 92 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
3Claude Sonnet 5 80.2% 71.4% to 88.9% 40 records over 10 items, pooled from 4 entries 80.7% alirezarezvani-a11y-audit (of 4 entries) $0.00368 run on subscription; shown at API list price for comparison 1,272 in, 114 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
4Claude Haiku 4.5 46.1% 34.1% to 58.1% 40 records over 10 items, pooled from 4 entries 62.8% rampstackco-accessibility-audit (of 4 entries) $0.00104 run on subscription; shown at API list price for comparison 765 in, 56 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Accessibility audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 46.1% interval does not reach the best $0.00104 run on subscription; shown at API list price for comparison $1.04
Claude Sonnet 5 80.2% interval overlaps the best $0.00368 run on subscription; shown at API list price for comparison $3.68
Claude Opus 5 87.1% interval overlaps the best $0.00704 run on subscription; shown at API list price for comparison $7.04
Claude Fable 5.1 87.9% the best score here $0.05126 run on subscription; shown at API list price for comparison $51.26

Good enough pick. Claude Sonnet 5 at 80.2% and $3.68 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Fable 5.1, the best on this table at 87.9% and $51.26.

tested in Claude Code, Fable 5, wider run, three ways

Accessibility audit, tested in Claude Code, Fable 5, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 89.7% 80.3% to 99.1% 40 records over 10 items, pooled from 4 entries 93.5% alirezarezvani-a11y-audit (of 4 entries) $0.04668 run on subscription; shown at API list price for comparison 1,160 in, 702 out 623 one pass, not measured 22.5 s 19.0 s to 27.1 s across the middle half of 120 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Accessibility audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5 89.7% the best score here $0.04668 run on subscription; shown at API list price for comparison $46.68

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Opus 5.5 series, wider run, three ways

Accessibility audit, tested in Claude Code, Opus 5.5 series, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 85.8% 74.4% to 97.2% 40 records over 10 items, pooled from 4 entries 86.9% alirezarezvani-a11y-audit (of 4 entries) $0.01591 run on subscription; shown at API list price for comparison 1,167 in, 562 out 472 one pass, not measured 18.3 s 16.7 s to 19.6 s across the middle half of 120 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Accessibility audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 85.8% the best score here $0.01591 run on subscription; shown at API list price for comparison $15.91

No good enough pick. One model ran this job this way, so there is nothing to pick between.

Code review · seeded code defects

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Wider run, three ways

Code review, tested via API, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5 mini 70.0% 59.2% to 80.7% 80 records over 10 items, pooled from 8 entries 73.8% jeffallan-code-reviewer (of 8 entries) not measured 665 in, 70 out not measured one pass, not measured not measured
2Gemini 3.1 Flash Lite 57.5% 40.1% to 74.9% 80 records over 10 items, pooled from 8 entries 60.2% wshobson-code-review-excellence (of 8 entries) not measured 783 in, 34 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 80 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Code review: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 57.5% interval overlaps the best not measured not measured
GPT-5 mini 70.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code, Wider run, three ways

Code review, tested in Claude Code, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5 93.5% 84.7% to 100.0%† 80 records over 10 items, pooled from 8 entries 95.5% addyosmani-code-review-and-quality (of 8 entries) $0.00948 run on subscription; shown at API list price for comparison 1,333 in, 113 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
2Claude Fable 5.1 89.0% 83.2% to 94.8% 80 records over 10 items, pooled from 8 entries 93.5% tied: jeffallan-code-reviewer, mattpocock-code-review, open-gsd-code-review (of 8 entries) $0.04548 run on subscription; shown at API list price for comparison 1,349 in, 640 out 555 one pass, not measured not measured
3Claude Sonnet 5 74.9% 62.4% to 87.4% 80 records over 10 items, pooled from 8 entries 83.3% rampstackco-code-review-web (of 8 entries) $0.00406 run on subscription; shown at API list price for comparison 1,659 in, 74 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
4Claude Haiku 4.5 58.3% 47.8% to 68.7% 80 records over 10 items, pooled from 8 entries 56.7% open-gsd-code-review (of 8 entries) $0.00120 run on subscription; shown at API list price for comparison 998 in, 41 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Code review: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 58.3% interval does not reach the best $0.00120 run on subscription; shown at API list price for comparison $1.20
Claude Sonnet 5 74.9% interval overlaps the best $0.00406 run on subscription; shown at API list price for comparison $4.06
Claude Opus 5 93.5% the best score here $0.00948 run on subscription; shown at API list price for comparison $9.48
Claude Fable 5.1 89.0% interval overlaps the best $0.04548 run on subscription; shown at API list price for comparison $45.48

Good enough pick. Claude Sonnet 5 at 74.9% and $4.06 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Opus 5, the best on this table at 93.5% and $9.48.

tested in Claude Code, Fable 5, wider run, three ways

Code review, tested in Claude Code, Fable 5, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 85.8% 77.4% to 94.2% 80 records over 10 items, pooled from 8 entries 91.8% open-gsd-code-review (of 8 entries) $0.04236 run on subscription; shown at API list price for comparison 1,547 in, 538 out 462 one pass, not measured 17.6 s 15.1 s to 21.8 s across the middle half of 240 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Code review: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5 85.8% the best score here $0.04236 run on subscription; shown at API list price for comparison $42.36

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Opus 5.5 series, wider run, three ways

Code review, tested in Claude Code, Opus 5.5 series, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 85.4% 78.3% to 92.5% 80 records over 10 items, pooled from 8 entries 88.2% tied: everyinc-ce-code-review, open-gsd-code-review (of 8 entries) $0.01243 run on subscription; shown at API list price for comparison 1,554 in, 311 out 232 one pass, not measured 15.0 s 13.4 s to 16.7 s across the middle half of 242 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Code review: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 85.4% the best score here $0.01243 run on subscription; shown at API list price for comparison $12.43

No good enough pick. One model ran this job this way, so there is nothing to pick between.

On-page audit · seeded on-page issues

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, First run, with and without the skill

On-page audit, tested via API, First run, with and without the skill. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Gemini 3.1 Flash Lite 48.8% 27.1% to 70.5% 20 records over 10 items, pooled from 2 entries 70.7% rampstackco-seo-onpage (of 2 entries) $0.00023 billed through the API 599 in, 53 out not measured one pass, not measured not measured
2GPT-5 mini 44.9% 26.7% to 63.1% 20 records over 10 items, pooled from 2 entries 49.2% rampstackco-seo-onpage (of 2 entries) $0.00029 billed through the API 566 in, 75 out not measured one pass, not measured not measured
3Claude Haiku 4.5 23.9% 1.8% to 46.1% 20 records over 10 items, pooled from 2 entries 47.4% rampstackco-seo-onpage (of 2 entries) $0.00085 billed through the API 667 in, 37 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 20 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

On-page audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 48.8% the best score here $0.00023 billed through the API $0.23
GPT-5 mini 44.9% interval overlaps the best $0.00029 billed through the API $0.29
Claude Haiku 4.5 23.9% interval overlaps the best $0.00085 billed through the API $0.85

Good enough pick. Gemini 3.1 Flash Lite at 48.8% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $0.23 for 1,000 tasks a month.

tested via API, Wider run, three ways

On-page audit, tested via API, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5 mini 52.2% 37.4% to 67.0% 30 records over 10 items, pooled from 3 entries 57.5% rampstackco-seo-onpage (of 3 entries) $0.00029 billed through the API 566 in, 76 out not measured one pass, not measured not measured
2Gemini 3.1 Flash Lite 48.8% 27.1% to 70.5% 30 records over 10 items, pooled from 3 entries 68.2% rampstackco-seo-onpage (of 3 entries) $0.00023 billed through the API 599 in, 53 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 30 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

0 of the 33 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

On-page audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 48.8% interval overlaps the best $0.00023 billed through the API $0.23
GPT-5 mini 52.2% the best score here $0.00029 billed through the API $0.29

Good enough pick. Gemini 3.1 Flash Lite at 48.8% and $0.23 for 1,000 tasks a month is the cheapest model whose bracket overlaps GPT-5 mini, the best on this table at 52.2% and $0.29.

tested in Claude Code, First run, single attempt

On-page audit, tested in Claude Code, First run, single attempt. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 92.3% 85.3% to 99.4% 20 records over 10 items, pooled from 2 entries 93.3% affaan-m-seo (of 2 entries) $0.03843 run on subscription; shown at API list price for comparison 1,068 in, 555 out 483 one pass, not measured not measured
2Claude Opus 5 87.9% 76.7% to 99.1% 20 records over 10 items, pooled from 2 entries 83.3% rampstackco-seo-onpage (of 2 entries) $0.00710 run on subscription; shown at API list price for comparison 1,068 in, 71 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

On-page audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5 87.9% interval overlaps the best $0.00710 run on subscription; shown at API list price for comparison $7.10
Claude Fable 5 92.3% the best score here $0.03843 run on subscription; shown at API list price for comparison $38.43

Good enough pick. Claude Opus 5 at 87.9% and $7.10 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Fable 5, the best on this table at 92.3% and $38.43.

tested in Claude Code, Second run, every task twice

On-page audit, tested in Claude Code, Second run, every task twice. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5 88.5% 77.6% to 99.5% 40 records over 10 items, pooled from 2 entries 83.3% rampstackco-seo-onpage (of 2 entries) not measured not measured in, 141 out 0 a gate holding at zero, not a measured zero 33 of 40 32.7 percentage points apart where not no sampling control here not measured
2Claude Fable 5 88.0% 79.2% to 96.9% 40 records over 10 items, pooled from 2 entries 94.0% affaan-m-seo (of 2 entries) not measured not measured in, 1,203 out 1,059 32 of 40 32.1 percentage points apart where not no sampling control here not measured
3Claude Sonnet 5 59.0% 46.0% to 72.0% 40 records over 10 items, pooled from 2 entries 64.7% affaan-m-seo (of 2 entries) not measured not measured in, 171 out 0 a gate holding at zero, not a measured zero 29 of 40 26.6 percentage points apart where not no sampling control here not measured
4Claude Haiku 4.5 23.1% 6.6% to 39.6% 40 records over 10 items, pooled from 2 entries 40.8% rampstackco-seo-onpage (of 2 entries) not measured not measured in, 83 out 0 a gate holding at zero, not a measured zero 32 of 40 36.2 percentage points apart where not no sampling control here not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

A cost per task needs both sides of the call. Only 0 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Only 0 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set. Why some figures are blank.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

On-page audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5 88.0% interval overlaps the best not measured not measured
Claude Haiku 4.5 23.1% interval does not reach the best not measured not measured
Claude Opus 5 88.5% the best score here not measured not measured
Claude Sonnet 5 59.0% interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code, Fable 5.1, single attempt

On-page audit, tested in Claude Code, Fable 5.1, single attempt. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5.1 98.3% 95.1% to 100.0%† 20 records over 10 items, pooled from 2 entries 95.0% tied: affaan-m-seo, rampstackco-seo-onpage (of 2 entries) $0.03907 run on subscription; shown at API list price for comparison 1,070 in, 568 out 499 one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

On-page audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5.1 98.3% the best score here $0.03907 run on subscription; shown at API list price for comparison $39.07

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Wider run, three ways

On-page audit, tested in Claude Code, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5.1 97.2% 93.5% to 100.0%† 30 records over 10 items, pooled from 3 entries 98.3% rampstackco-seo-onpage (of 3 entries) $0.03873 run on subscription; shown at API list price for comparison 1,070 in, 561 out 491 one pass, not measured not measured
2Claude Opus 5 88.1% 76.9% to 99.2% 30 records over 10 items, pooled from 3 entries 83.3% alirezarezvani-seo-audit (of 3 entries) $0.00715 run on subscription; shown at API list price for comparison 1,068 in, 72 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
3Claude Sonnet 5 53.0% 43.8% to 62.2% 30 records over 10 items, pooled from 3 entries 66.5% affaan-m-seo (of 3 entries) $0.00362 run on subscription; shown at API list price for comparison 1,394 in, 84 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
4Claude Haiku 4.5 23.1% 3.6% to 42.6% 30 records over 10 items, pooled from 3 entries 41.1% rampstackco-seo-onpage (of 3 entries) $0.00105 run on subscription; shown at API list price for comparison 832 in, 43 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

On-page audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 23.1% interval does not reach the best $0.00105 run on subscription; shown at API list price for comparison $1.05
Claude Sonnet 5 53.0% interval does not reach the best $0.00362 run on subscription; shown at API list price for comparison $3.62
Claude Opus 5 88.1% interval overlaps the best $0.00715 run on subscription; shown at API list price for comparison $7.15
Claude Fable 5.1 97.2% the best score here $0.03873 run on subscription; shown at API list price for comparison $38.73

Good enough pick. Claude Opus 5 at 88.1% and $7.15 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Fable 5.1, the best on this table at 97.2% and $38.73.

tested in Claude Code, Fable 5, wider run, three ways

On-page audit, tested in Claude Code, Fable 5, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 93.2% 86.4% to 100.0%† 30 records over 10 items, pooled from 3 entries 94.4% alirezarezvani-seo-audit (of 3 entries) $0.03795 run on subscription; shown at API list price for comparison 1,282 in, 503 out 431 one pass, not measured 17.8 s 16.4 s to 19.0 s across the middle half of 90 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

On-page audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5 93.2% the best score here $0.03795 run on subscription; shown at API list price for comparison $37.96

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Opus 5.5 series, wider run, three ways

On-page audit, tested in Claude Code, Opus 5.5 series, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 93.3% 86.1% to 100.0%† 30 records over 10 items, pooled from 3 entries 98.3% rampstackco-seo-onpage (of 3 entries) $0.01317 run on subscription; shown at API list price for comparison 1,289 in, 401 out 334 one pass, not measured 15.3 s 14.4 s to 16.2 s across the middle half of 90 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

On-page audit: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 93.3% the best score here $0.01317 run on subscription; shown at API list price for comparison $13.17

No good enough pick. One model ran this job this way, so there is nothing to pick between.

Skill authoring · specification requirements

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, First run, with and without the skill

Skill authoring, tested via API, First run, with and without the skill. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Gemini 3.1 Flash Lite 87.3% 80.1% to 94.4% 20 records over 10 items, pooled from 2 entries 100.0% tied: anthropics-skill-creator, rampstackco-skill-creation-walkthrough (of 2 entries) $0.00088 billed through the API 59 in, 580 out not measured one pass, not measured not measured
2GPT-5 mini 73.2% 60.9% to 85.5% 20 records over 10 items, pooled from 2 entries 100.0% tied: anthropics-skill-creator, rampstackco-skill-creation-walkthrough (of 2 entries) $0.00297 billed through the API 61 in, 1,478 out not measured one pass, not measured not measured
3Claude Haiku 4.5 61.8% 52.3% to 71.3% 20 records over 10 items, pooled from 2 entries 100.0% tied: anthropics-skill-creator, rampstackco-skill-creation-walkthrough (of 2 entries) $0.00618 billed through the API 68 in, 1,222 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 20 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Skill authoring: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 87.3% the best score here $0.00088 billed through the API $0.88
GPT-5 mini 73.2% interval overlaps the best $0.00297 billed through the API $2.97
Claude Haiku 4.5 61.8% interval does not reach the best $0.00618 billed through the API $6.18

Good enough pick. Gemini 3.1 Flash Lite at 87.3% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $0.88 for 1,000 tasks a month.

tested via API, Wider run, three ways

Skill authoring, tested via API, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Gemini 3.1 Flash Lite 87.3% 80.1% to 94.4% 40 records over 10 items, pooled from 4 entries 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, rampstackco-skill-creation-walkthrough, yusufkaraaslan-skill-builder (of 4 entries) $0.00088 billed through the API 59 in, 580 out not measured one pass, not measured not measured
2GPT-5 mini 64.8% 56.1% to 73.4% 40 records over 10 items, pooled from 4 entries 100.0% tied: anthropics-skill-creator, rampstackco-skill-creation-walkthrough (of 4 entries) $0.00297 billed through the API 64 in, 1,610 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 40 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

0 of the 42 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Skill authoring: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 87.3% the best score here $0.00088 billed through the API $0.88
GPT-5 mini 64.8% interval does not reach the best $0.00297 billed through the API $2.97

Good enough pick. Gemini 3.1 Flash Lite at 87.3% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $0.88 for 1,000 tasks a month.

tested in Claude Code, First run, single attempt

Skill authoring, tested in Claude Code, First run, single attempt. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 100.0% 100.0% to 100.0% 20 records over 10 items, pooled from 2 entries 100.0% anthropics-skill-creator (of 2 entries) $0.10060 run on subscription; shown at API list price for comparison 313 in, 1,949 out 64 one pass, not measured not measured
1Claude Opus 5 100.0% 100.0% to 100.0% 20 records over 10 items, pooled from 2 entries 100.0% tied: anthropics-skill-creator, rampstackco-skill-creation-walkthrough (of 2 entries) $0.06061 run on subscription; shown at API list price for comparison 313 in, 2,362 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Skill authoring: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5 100.0% the best score here $0.06061 run on subscription; shown at API list price for comparison $60.61
Claude Fable 5 100.0% the best score here $0.10060 run on subscription; shown at API list price for comparison $100.60

Good enough pick. Claude Opus 5 at 100.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $60.61 for 1,000 tasks a month.

tested in Claude Code, Second run, every task twice

Skill authoring, tested in Claude Code, Second run, every task twice. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 100.0% 100.0% to 100.0% 22 records over 10 items, pooled from 2 entries 100.0% anthropics-skill-creator (of 2 entries) $0.19551 run on subscription; shown at API list price for comparison 628 in, 3,785 out 131 19 of 21 72.7 percentage points apart where not 1 more answered once no sampling control here not measured
1Claude Opus 5 100.0% 100.0% to 100.0% 30 records over 10 items, pooled from 2 entries 100.0% rampstackco-skill-creation-walkthrough (of 2 entries) $0.12485 run on subscription; shown at API list price for comparison 627 in, 4,869 out 0 a gate holding at zero, not a measured zero 28 of 29 72.7 percentage points apart where not no sampling control here not measured
1Claude Sonnet 5 100.0% 100.0% to 100.0% 40 records over 10 items, pooled from 2 entries 92.7% anthropics-skill-creator (of 2 entries) $0.03975 run on subscription; shown at API list price for comparison 1,278 in, 3,719 out 0 a gate holding at zero, not a measured zero 35 of 40 72.7 percentage points apart where not no sampling control here not measured
2Claude Haiku 4.5 70.0% 63.5% to 76.5% 40 records over 10 items, pooled from 2 entries 100.0% tied: anthropics-skill-creator, rampstackco-skill-creation-walkthrough (of 2 entries) $0.00996 run on subscription; shown at API list price for comparison 467 in, 1,898 out 0 a gate holding at zero, not a measured zero 29 of 40 36.4 percentage points apart where not no sampling control here not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Skill authoring: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 70.0% interval does not reach the best $0.00996 run on subscription; shown at API list price for comparison $9.96
Claude Sonnet 5 100.0% the best score here $0.03975 run on subscription; shown at API list price for comparison $39.75
Claude Opus 5 100.0% the best score here $0.12485 run on subscription; shown at API list price for comparison $124.85
Claude Fable 5 100.0% the best score here $0.19551 run on subscription; shown at API list price for comparison $195.51

Good enough pick. Claude Sonnet 5 at 100.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $39.75 for 1,000 tasks a month.

tested in Claude Code, Fable 5.1, single attempt

Skill authoring, tested in Claude Code, Fable 5.1, single attempt. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5.1 100.0% 100.0% to 100.0% 20 records over 10 items, pooled from 2 entries 100.0% rampstackco-skill-creation-walkthrough (of 2 entries) $0.15512 run on subscription; shown at API list price for comparison 315 in, 3,039 out 145 one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Skill authoring: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5.1 100.0% the best score here $0.15512 run on subscription; shown at API list price for comparison $155.12

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Wider run, three ways

Skill authoring, tested in Claude Code, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5.1 100.0% 100.0% to 100.0% 40 records over 10 items, pooled from 4 entries 100.0% tied: openclaw-skill-creator, yusufkaraaslan-skill-builder (of 4 entries) $0.16037 run on subscription; shown at API list price for comparison 316 in, 3,144 out 137 one pass, not measured not measured
1Claude Opus 5 100.0% 100.0% to 100.0% 40 records over 10 items, pooled from 4 entries 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, rampstackco-skill-creation-walkthrough, yusufkaraaslan-skill-builder (of 4 entries) $0.05962 run on subscription; shown at API list price for comparison 313 in, 2,322 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
1Claude Sonnet 5 100.0% 100.0% to 100.0% 40 records over 10 items, pooled from 4 entries 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, yusufkaraaslan-skill-builder (of 4 entries) $0.02027 run on subscription; shown at API list price for comparison 639 in, 1,899 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
2Claude Haiku 4.5 67.3% 59.7% to 74.9% 40 records over 10 items, pooled from 4 entries 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, rampstackco-skill-creation-walkthrough, yusufkaraaslan-skill-builder (of 4 entries) $0.00504 run on subscription; shown at API list price for comparison 233 in, 961 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Skill authoring: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 67.3% interval does not reach the best $0.00504 run on subscription; shown at API list price for comparison $5.04
Claude Sonnet 5 100.0% the best score here $0.02027 run on subscription; shown at API list price for comparison $20.27
Claude Opus 5 100.0% the best score here $0.05962 run on subscription; shown at API list price for comparison $59.62
Claude Fable 5.1 100.0% the best score here $0.16037 run on subscription; shown at API list price for comparison $160.37

Good enough pick. Claude Sonnet 5 at 100.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $20.27 for 1,000 tasks a month.

tested in Claude Code, Fable 5, wider run, three ways

Skill authoring, tested in Claude Code, Fable 5, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 100.0% 100.0% to 100.0% 40 records over 10 items, pooled from 4 entries 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, yusufkaraaslan-skill-builder (of 4 entries) $0.09942 run on subscription; shown at API list price for comparison 527 in, 1,883 out 34 one pass, not measured 44.6 s 37.8 s to 54.1 s across the middle half of 120 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Skill authoring: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5 100.0% the best score here $0.09942 run on subscription; shown at API list price for comparison $99.42

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Opus 5.5 series, wider run, three ways

Skill authoring, tested in Claude Code, Opus 5.5 series, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 99.8% 99.3% to 100.0%† 40 records over 10 items, pooled from 4 entries 100.0% tied: anthropics-skill-creator, openclaw-skill-creator, rampstackco-skill-creation-walkthrough, yusufkaraaslan-skill-builder (of 4 entries) $0.07276 run on subscription; shown at API list price for comparison 534 in, 3,531 out 124 one pass, not measured 52.8 s 43.8 s to 67.1 s across the middle half of 120 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Skill authoring: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 99.8% the best score here $0.07276 run on subscription; shown at API list price for comparison $72.76

No good enough pick. One model ran this job this way, so there is nothing to pick between.

Spec writing · required sections and acceptance criteria

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, First run, with and without the skill

Spec writing, tested via API, First run, with and without the skill. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 99.5% 98.7% to 100.0%† 20 records over 10 items, pooled from 2 entries 99.1% rampstackco-pm-spec-writing (of 2 entries) $0.00332 billed through the API 118 in, 641 out not measured one pass, not measured not measured
2Gemini 3.1 Flash Lite 99.1% 97.3% to 100.0%† 20 records over 10 items, pooled from 2 entries 99.1% tied: affaan-m-product-capability, rampstackco-pm-spec-writing (of 2 entries) $0.00080 billed through the API 106 in, 513 out not measured one pass, not measured not measured
3GPT-5 mini 94.5% 92.0% to 97.1% 20 records over 10 items, pooled from 2 entries 90.0% affaan-m-product-capability (of 2 entries) $0.00248 billed through the API 107 in, 1,227 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

0 of the 40 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Spec writing: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 99.1% interval overlaps the best $0.00080 billed through the API $0.80
GPT-5 mini 94.5% interval does not reach the best $0.00248 billed through the API $2.48
Claude Haiku 4.5 99.5% the best score here $0.00332 billed through the API $3.32

Good enough pick. Gemini 3.1 Flash Lite at 99.1% and $0.80 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Haiku 4.5, the best on this table at 99.5% and $3.32.

tested via API, Wider run, three ways

Spec writing, tested via API, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Gemini 3.1 Flash Lite 99.1% 97.3% to 100.0%† 60 records over 10 items, pooled from 6 entries 100.0% tied: addyosmani-spec-driven-development, alirezarezvani-spec-driven-workflow, github-prd, phuryn-create-prd (of 6 entries) $0.00080 billed through the API 106 in, 513 out not measured one pass, not measured not measured
2GPT-5 mini 96.4% 95.0% to 97.7% 60 records over 10 items, pooled from 6 entries 97.3% affaan-m-product-capability (of 6 entries) $0.00248 billed through the API 107 in, 1,204 out not measured one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

0 of the 60 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Spec writing: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Gemini 3.1 Flash Lite 99.1% the best score here $0.00080 billed through the API $0.80
GPT-5 mini 96.4% interval overlaps the best $0.00248 billed through the API $2.48

Good enough pick. Gemini 3.1 Flash Lite at 99.1% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $0.80 for 1,000 tasks a month.

tested in Claude Code, First run, single attempt

Spec writing, tested in Claude Code, First run, single attempt. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 93.2% 91.2% to 95.2% 20 records over 10 items, pooled from 2 entries 93.6% affaan-m-product-capability (of 2 entries) $0.08369 run on subscription; shown at API list price for comparison 386 in, 1,597 out 20 one pass, not measured not measured
2Claude Opus 5 92.3% 90.9% to 93.6% 20 records over 10 items, pooled from 2 entries 93.6% rampstackco-pm-spec-writing (of 2 entries) $0.05944 run on subscription; shown at API list price for comparison 386 in, 2,300 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Spec writing: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5 92.3% interval overlaps the best $0.05944 run on subscription; shown at API list price for comparison $59.44
Claude Fable 5 93.2% the best score here $0.08369 run on subscription; shown at API list price for comparison $83.69

Good enough pick. Claude Opus 5 at 92.3% and $59.44 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Fable 5, the best on this table at 93.2% and $83.69.

tested in Claude Code, Second run, every task twice

Spec writing, tested in Claude Code, Second run, every task twice. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 98.6% 97.7% to 99.6% 40 records over 10 items, pooled from 2 entries 97.7% rampstackco-pm-spec-writing (of 2 entries) not measured not measured in, 1,145 out 0 a gate holding at zero, not a measured zero 26 of 40 9.1 percentage points apart where not no sampling control here not measured
2Claude Fable 5 95.7% 94.6% to 96.7% 40 records over 10 items, pooled from 2 entries 92.7% tied: affaan-m-product-capability, rampstackco-pm-spec-writing (of 2 entries) not measured not measured in, 3,234 out 38 25 of 40 9.1 percentage points apart where not no sampling control here not measured
3Claude Sonnet 5 94.8% 93.0% to 96.5% 40 records over 10 items, pooled from 2 entries 95.0% rampstackco-pm-spec-writing (of 2 entries) not measured not measured in, 2,948 out 0 a gate holding at zero, not a measured zero 21 of 40 9.1 percentage points apart where not no sampling control here not measured
4Claude Opus 5 92.5% 91.3% to 93.7% 40 records over 10 items, pooled from 2 entries 94.5% rampstackco-pm-spec-writing (of 2 entries) not measured not measured in, 5,009 out 0 a gate holding at zero, not a measured zero 27 of 40 9.8 percentage points apart where not no sampling control here not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

A cost per task needs both sides of the call. Only 26 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set.

A cost per task needs both sides of the call. Only 30 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set.

A cost per task needs both sides of the call. Only 32 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set.

A cost per task needs both sides of the call. Only 34 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Only 26 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set. Why some figures are blank.

Only 30 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set. Why some figures are blank.

Only 32 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set. Why some figures are blank.

Only 34 of the 40 runs behind this figure reported an input total, so there is no way to total it up. The figure is left blank rather than guessed from the runs that did report it: those runs are not a random half of the set. Why some figures are blank.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Spec writing: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5 95.7% interval does not reach the best not measured not measured
Claude Haiku 4.5 98.6% the best score here not measured not measured
Claude Opus 5 92.5% interval does not reach the best not measured not measured
Claude Sonnet 5 94.8% interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code, Fable 5.1, single attempt

Spec writing, tested in Claude Code, Fable 5.1, single attempt. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5.1 97.7% 96.2% to 99.2% 20 records over 10 items, pooled from 2 entries 99.1% rampstackco-pm-spec-writing (of 2 entries) $0.11240 run on subscription; shown at API list price for comparison 388 in, 2,170 out 85 one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Spec writing: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5.1 97.7% the best score here $0.11240 run on subscription; shown at API list price for comparison $112.40

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Wider run, three ways

Spec writing, tested in Claude Code, Wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 99.2% 98.7% to 99.7% 60 records over 10 items, pooled from 6 entries 99.1% github-prd (of 6 entries) $0.00315 run on subscription; shown at API list price for comparison 283 in, 574 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
2Claude Fable 5.1 97.9% 96.8% to 99.0% 60 records over 10 items, pooled from 6 entries 100.0% rampstackco-pm-spec-writing (of 6 entries) $0.10719 run on subscription; shown at API list price for comparison 388 in, 2,066 out 110 one pass, not measured not measured
3Claude Sonnet 5 96.2% 94.9% to 97.6% 60 records over 10 items, pooled from 6 entries 97.3% github-prd (of 6 entries) $0.01601 run on subscription; shown at API list price for comparison 712 in, 1,459 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured
4Claude Opus 5 92.3% 91.1% to 93.5% 60 records over 10 items, pooled from 6 entries 94.5% phuryn-create-prd (of 6 entries) $0.06098 run on subscription; shown at API list price for comparison 386 in, 2,362 out 0 a gate holding at zero, not a measured zero one pass, not measured not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Spec writing: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 99.2% the best score here $0.00315 run on subscription; shown at API list price for comparison $3.15
Claude Sonnet 5 96.2% interval does not reach the best $0.01601 run on subscription; shown at API list price for comparison $16.01
Claude Opus 5 92.3% interval does not reach the best $0.06098 run on subscription; shown at API list price for comparison $60.98
Claude Fable 5.1 97.9% interval overlaps the best $0.10719 run on subscription; shown at API list price for comparison $107.19

Good enough pick. Claude Haiku 4.5 at 99.2% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $3.15 for 1,000 tasks a month.

tested in Claude Code, Fable 5, wider run, three ways

Spec writing, tested in Claude Code, Fable 5, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 95.9% 94.7% to 97.1% 60 records over 10 items, pooled from 6 entries 96.4% phuryn-create-prd (of 6 entries) $0.08887 run on subscription; shown at API list price for comparison 610 in, 1,655 out 13 one pass, not measured 36.7 s 33.0 s to 41.6 s across the middle half of 180 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Spec writing: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Fable 5 95.9% the best score here $0.08887 run on subscription; shown at API list price for comparison $88.87

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code, Opus 5.5 series, wider run, three ways

Spec writing, tested in Claude Code, Opus 5.5 series, wider run, three ways. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 99.5% 98.9% to 100.0%† 60 records over 10 items, pooled from 6 entries 100.0% tied: affaan-m-product-capability, github-prd, phuryn-create-prd, rampstackco-pm-spec-writing (of 6 entries) $0.04784 run on subscription; shown at API list price for comparison 607 in, 2,271 out 56 one pass, not measured 36.0 s 32.9 s to 49.2 s across the middle half of 180 calls

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Spec writing: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 99.5% the best score here $0.04784 run on subscription; shown at API list price for comparison $47.84

No good enough pick. One model ran this job this way, so there is nothing to pick between.

Prompting tips, single-turn

Better step-by-step answers

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Better step-by-step answers, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Gemini 3.1 Flash Lite 40.0% 26.3% to 53.7% 50 records 50.0% C05-think-step-by-step not measured 220 in, 11 out not measured 20 of 20 every repeat agreed at temperature 0 asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
2Claude Haiku 4.5 30.0% 17.2% to 42.8% 50 records 50.0% C05-think-step-by-step not measured 273 in, 25 out not measured 20 of 20 every repeat agreed at temperature 0 not measured
3GPT-5 mini 16.0% 5.7% to 26.3% 50 records 50.0% C05-think-step-by-step not measured 237 in, 53 out not measured 17 of 20 40.0 percentage points apart where not at the vendor default asked five times: 10 of 10 the same every time, 72 to 100 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 50 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Better step-by-step answers: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 30.0% interval overlaps the best not measured not measured
Gemini 3.1 Flash Lite 40.0% the best score here not measured not measured
GPT-5 mini 16.0% interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code

Better step-by-step answers, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 40.0% 26.3% to 53.7% 50 records 50.0% C05-think-step-by-step $0.02052 run on subscription; shown at API list price for comparison 1,444 in, 122 out 107 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
1Claude Fable 5.1 40.0% 26.3% to 53.7% 50 records 50.0% C05-think-step-by-step $0.02103 run on subscription; shown at API list price for comparison 1,454 in, 130 out 0 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
1Claude Opus 5 40.0% 26.3% to 53.7% 50 records 46.0% C05-think-step-by-step $0.00807 run on subscription; shown at API list price for comparison 1,444 in, 34 out 0 a gate holding at zero, not a measured zero 19 of 20 60.0 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
1Claude Sonnet 5 40.0% 26.3% to 53.7% 50 records 50.0% C05-think-step-by-step $0.00630 run on subscription; shown at API list price for comparison 3,074 in, 15 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip
2Claude Haiku 4.5 20.0% 8.8% to 31.2% 50 records 50.0% C05-think-step-by-step $0.00135 run on subscription; shown at API list price for comparison 1,098 in, 50 out 0 a gate holding at zero, not a measured zero 17 of 20 46.7 percentage points apart where not no sampling control here asked five times: 8 of 10 the same every time, 49 to 94 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Better step-by-step answers: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 20.0% interval overlaps the best $0.00135 run on subscription; shown at API list price for comparison $1.35
Claude Sonnet 5 40.0% the best score here $0.00630 run on subscription; shown at API list price for comparison $6.30
Claude Opus 5 40.0% the best score here $0.00807 run on subscription; shown at API list price for comparison $8.07
Claude Fable 5 40.0% the best score here $0.02052 run on subscription; shown at API list price for comparison $20.53
Claude Fable 5.1 40.0% the best score here $0.02103 run on subscription; shown at API list price for comparison $21.03

Good enough pick. Claude Haiku 4.5 at 20.0% and $1.35 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Sonnet 5, the best on this table at 40.0% and $6.30.

tested in Claude Code, Opus 5.5 series, each task asked once

Better step-by-step answers, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 40.0% 8.0% to 72.0% 10 items 50.0% C05-think-step-by-step $0.00234 run on subscription; shown at API list price for comparison 509 in, 15 out 3 one pass, not measured asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here 12.1 s 11.9 s to 12.2 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Better step-by-step answers: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 40.0% the best score here $0.00234 run on subscription; shown at API list price for comparison $2.34

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Better step-by-step answers, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-6 Sol 40.0% interval not measured 10 items 40.0% C05-think-step-by-step not measured 6,680 in, 5 out 0 one pass, not measured 8.2 s 7.5 s to 8.9 s across the middle half of 20 calls
2GPT-5.4 mini 30.0% interval not measured 10 items 40.0% C05-think-step-by-step not measured 3,879 in, 5 out 0 one pass, not measured not measured
2GPT-5.6 Luna 30.0% interval not measured 10 items 40.0% C05-think-step-by-step not measured 4,599 in, 5 out 0 one pass, not measured not measured
2GPT-5.6 Terra 30.0% interval not measured 10 items 40.0% C05-think-step-by-step not measured 6,135 in, 5 out 0 one pass, not measured not measured
2GPT-6 Luna 30.0% interval not measured 10 items 40.0% C05-think-step-by-step not measured 6,680 in, 5 out 0 one pass, not measured 8.1 s 7.8 s to 8.8 s across the middle half of 20 calls

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Better step-by-step answers: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 30.0% interval does not reach the best not measured not measured
GPT-5.6 Luna 30.0% interval does not reach the best not measured not measured
GPT-5.6 Terra 30.0% interval does not reach the best not measured not measured
GPT-6 Luna 30.0% interval does not reach the best not measured not measured
GPT-6 Sol 40.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Getting clean JSON back

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Getting clean JSON back, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 0.0% 0.0% to 0.0% 50 records 100.0% C01-json-schema not measured 277 in, 415 out not measured 20 of 20 every repeat agreed at temperature 0 not measured
1Gemini 3.1 Flash Lite 0.0% 0.0% to 0.0% 50 records 100.0% C01-json-schema not measured 219 in, 386 out not measured 20 of 20 every repeat agreed at temperature 0 asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
1GPT-5 mini 0.0% 0.0% to 0.0% 50 records 100.0% C01-json-schema not measured 235 in, 393 out not measured 20 of 20 every repeat agreed at the vendor default asked five times: 10 of 10 the same every time, 72 to 100 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 50 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting clean JSON back: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 0.0% the best score here not measured not measured
Gemini 3.1 Flash Lite 0.0% the best score here not measured not measured
GPT-5 mini 0.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code

Getting clean JSON back, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 0.0% 0.0% to 0.0% 50 records 100.0% C01-json-schema $0.04321 run on subscription; shown at API list price for comparison 1,496 in, 565 out 46 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
1Claude Fable 5.1 0.0% 0.0% to 0.0% 50 records 100.0% C01-json-schema $0.04114 run on subscription; shown at API list price for comparison 1,506 in, 522 out 0 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
1Claude Haiku 4.5 0.0% 0.0% to 0.0% 50 records 100.0% C01-json-schema $0.00316 run on subscription; shown at API list price for comparison 1,102 in, 412 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
1Claude Opus 5 0.0% 0.0% to 0.0% 50 records 100.0% C01-json-schema $0.02287 run on subscription; shown at API list price for comparison 1,496 in, 616 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
1Claude Sonnet 5 0.0% 0.0% to 0.0% 50 records 100.0% C01-json-schema $0.01160 run on subscription; shown at API list price for comparison 3,126 in, 535 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting clean JSON back: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 0.0% the best score here $0.00316 run on subscription; shown at API list price for comparison $3.16
Claude Sonnet 5 0.0% the best score here $0.01160 run on subscription; shown at API list price for comparison $11.60
Claude Opus 5 0.0% the best score here $0.02287 run on subscription; shown at API list price for comparison $22.87
Claude Fable 5.1 0.0% the best score here $0.04114 run on subscription; shown at API list price for comparison $41.14
Claude Fable 5 0.0% the best score here $0.04321 run on subscription; shown at API list price for comparison $43.21

Good enough pick. Claude Haiku 4.5 at 0.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $3.16 for 1,000 tasks a month.

tested in Claude Code, Opus 5.5 series, each task asked once

Getting clean JSON back, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 0.0% 0.0% to 0.0% 10 items 100.0% C01-json-schema $0.00438 run on subscription; shown at API list price for comparison 519 in, 115 out 0 one pass, not measured asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here 12.5 s 12.1 s to 13.6 s across the middle half of 40 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting clean JSON back: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 0.0% the best score here $0.00438 run on subscription; shown at API list price for comparison $4.38

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Getting clean JSON back, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 0.0% interval not measured 10 items 100.0% C01-json-schema not measured 3,879 in, 49 out 0 one pass, not measured not measured
1GPT-5.6 Luna 0.0% interval not measured 10 items 100.0% C01-json-schema not measured 4,620 in, 53 out 0 one pass, not measured not measured
1GPT-5.6 Terra 0.0% interval not measured 10 items 100.0% C01-json-schema not measured 6,135 in, 44 out 0 one pass, not measured not measured
1GPT-6 Luna 0.0% interval not measured 10 items 100.0% C01-json-schema not measured 6,679 in, 51 out 0 one pass, not measured 9.5 s 8.2 s to 10.7 s across the middle half of 20 calls
1GPT-6 Sol 0.0% interval not measured 10 items 100.0% C01-json-schema not measured 6,659 in, 44 out 0 one pass, not measured 8.4 s 8.0 s to 10.5 s across the middle half of 20 calls

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting clean JSON back: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 0.0% the best score here not measured not measured
GPT-5.6 Luna 0.0% the best score here not measured not measured
GPT-5.6 Terra 0.0% the best score here not measured not measured
GPT-6 Luna 0.0% the best score here not measured not measured
GPT-6 Sol 0.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Getting the length right

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Getting the length right, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Gemini 3.1 Flash Lite 31.0% 5.8% to 56.1% 10 items 99.4% C17-exact-length not measured 120 in, 3,398 out not measured 20 of 20 every repeat agreed temperature 0 asked for asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
2Claude Haiku 4.5 24.7% 1.2% to 48.2% 10 items 94.8% C17-exact-length not measured 164 in, 2,238 out not measured 10 of 20 2.3 percentage points apart where not temperature 0 asked for not measured
3GPT-5 mini 1.6% 0.0% to 4.6%† 10 items 96.8% C17-exact-length not measured 139 in, 5,014 out not measured 8 of 20 4.9 percentage points apart where not temperature 0 asked for asked five times: 8 of 10 the same every time, 49 to 94 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

0 of the 70 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting the length right: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 24.7% interval overlaps the best not measured not measured
Gemini 3.1 Flash Lite 31.0% the best score here not measured not measured
GPT-5 mini 1.6% interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested via API, API twin for the Codex run

Getting the length right, tested via API, API twin for the Codex run. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 15.7% interval not measured 10 items 96.9% C17-exact-length $0.01442 billed through the API 139 in, 3,182 out not measured 8 of 20 3.5 percentage points apart where not at temperature 0 not measured

0 of the 70 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting the length right: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 15.7% the best score here $0.01442 billed through the API $14.42

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code

Getting the length right, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 76.0% 53.7% to 98.4% 10 items 99.5% C17-exact-length $0.16668 run on subscription; shown at API list price for comparison 1,768 in, 2,980 out 437 7 of 20 7.1 percentage points apart where not no sampling control here asked five times: 1 of 10 the same every time, 2 to 40 percent, no sampling control here not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
2Claude Fable 5.1 50.2% 21.9% to 78.5% 10 items 99.5% C17-exact-length $0.22184 run on subscription; shown at API list price for comparison 1,786 in, 4,080 out 531 9 of 20 9.7 percentage points apart where not no sampling control here asked five times: 4 of 10 the same every time, 17 to 69 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
3Claude Haiku 4.5 27.5% 0.0% to 54.9% 10 items 92.5% C17-exact-length $0.01432 run on subscription; shown at API list price for comparison 1,319 in, 2,599 out 0 a gate holding at zero, not a measured zero 7 of 20 4.6 percentage points apart where not no sampling control here asked five times: 7 of 10 the same every time, 40 to 89 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
4Claude Sonnet 5 17.8% 0.0% to 41.0%† 10 items 95.2% C17-exact-length $0.06459 run on subscription; shown at API list price for comparison 4,050 in, 5,649 out 0 a gate holding at zero, not a measured zero 8 of 20 5.4 percentage points apart where not no sampling control here asked five times: 7 of 10 the same every time, 40 to 89 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip
5Claude Opus 5 15.0% 0.0% to 35.1%† 10 items 97.3% C17-exact-length $0.18907 run on subscription; shown at API list price for comparison 1,768 in, 7,209 out 0 a gate holding at zero, not a measured zero 7 of 20 5.0 percentage points apart where not no sampling control here asked five times: 8 of 10 the same every time, 49 to 94 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting the length right: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 27.5% interval overlaps the best $0.01432 run on subscription; shown at API list price for comparison $14.32
Claude Sonnet 5 17.8% interval does not reach the best $0.06459 run on subscription; shown at API list price for comparison $64.59
Claude Fable 5 76.0% the best score here $0.16668 run on subscription; shown at API list price for comparison $166.69
Claude Opus 5 15.0% interval does not reach the best $0.18907 run on subscription; shown at API list price for comparison $189.07
Claude Fable 5.1 50.2% interval overlaps the best $0.22184 run on subscription; shown at API list price for comparison $221.84

Good enough pick. Claude Haiku 4.5 at 27.5% and $14.32 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Fable 5, the best on this table at 76.0% and $166.69.

tested in Claude Code, Opus 5.5 series, each task asked once

Getting the length right, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 80.1% 62.5% to 97.8% 10 items 100.0% C17-exact-length $0.01233 run on subscription; shown at API list price for comparison 473 in, 522 out 163 one pass, not measured asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here 16.3 s 15.0 s to 19.9 s across the middle half of 40 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting the length right: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 80.1% the best score here $0.01233 run on subscription; shown at API list price for comparison $12.33

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, replaying two earlier tests

Getting the length right, tested in Codex, Codex run, replaying two earlier tests. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 18.6% interval not measured 10 items 96.7% C17-exact-length not measured 45,193 in, 2,211 out 0 6 of 20 6.4 percentage points apart where not setting not recorded not measured

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting the length right: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 18.6% the best score here not measured not measured

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Getting the length right, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.6 Terra 62.4% interval not measured 10 items 91.4% C17-exact-length not measured 6,108 in, 160 out 0 one pass, not measured not measured
2GPT-5.6 Luna 57.7% interval not measured 10 items 94.6% C17-exact-length not measured 4,593 in, 154 out 0 one pass, not measured not measured
3GPT-6 Luna 51.0% interval not measured 10 items 91.9% C17-exact-length not measured 6,652 in, 172 out 0 one pass, not measured 11.2 s 9.5 s to 14.4 s across the middle half of 20 calls
4GPT-5.4 mini 20.6% interval not measured 10 items 95.4% C17-exact-length not measured 3,851 in, 326 out 0 one pass, not measured not measured

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Getting the length right: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 20.6% interval does not reach the best not measured not measured
GPT-5.6 Luna 57.7% interval does not reach the best not measured not measured
GPT-5.6 Terra 62.4% the best score here not measured not measured
GPT-6 Luna 51.0% interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Instructions before or after

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Instructions before or after, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 100.0% 100.0% to 100.0% 50 records 100.0% C02-instruction-after not measured 981 in, 41 out not measured 20 of 20 every repeat agreed at temperature 0 not measured
1Gemini 3.1 Flash Lite 100.0% 100.0% to 100.0% 50 records 100.0% C02-instruction-after not measured 986 in, 41 out not measured 20 of 20 every repeat agreed at temperature 0 asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
1GPT-5 mini 100.0% 100.0% to 100.0% 50 records 100.0% C02-instruction-after not measured 900 in, 71 out not measured 20 of 20 every repeat agreed at the vendor default asked five times: 10 of 10 the same every time, 72 to 100 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 50 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Instructions before or after: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 100.0% the best score here not measured not measured
Gemini 3.1 Flash Lite 100.0% the best score here not measured not measured
GPT-5 mini 100.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code

Instructions before or after, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 100.0% 100.0% to 100.0% 50 records 100.0% C02-instruction-after $0.02645 run on subscription; shown at API list price for comparison 2,465 in, 36 out 0 20 of 20 every repeat agreed no sampling control here asked five times: withheld: 9 of 10 tasks got five usable answers not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
1Claude Fable 5.1 100.0% 100.0% to 100.0% 50 records 100.0% C02-instruction-after $0.02928 run on subscription; shown at API list price for comparison 2,748 in, 36 out 0 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
1Claude Haiku 4.5 100.0% 100.0% to 100.0% 50 records 100.0% C02-instruction-after $0.00201 run on subscription; shown at API list price for comparison 1,806 in, 41 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
1Claude Opus 5 100.0% 100.0% to 100.0% 50 records 100.0% C02-instruction-after $0.01323 run on subscription; shown at API list price for comparison 2,465 in, 36 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
1Claude Sonnet 5 100.0% 100.0% to 100.0% 50 records 100.0% C02-instruction-after $0.00855 run on subscription; shown at API list price for comparison 4,095 in, 36 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Instructions before or after: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 100.0% the best score here $0.00201 run on subscription; shown at API list price for comparison $2.01
Claude Sonnet 5 100.0% the best score here $0.00855 run on subscription; shown at API list price for comparison $8.55
Claude Opus 5 100.0% the best score here $0.01323 run on subscription; shown at API list price for comparison $13.23
Claude Fable 5 100.0% the best score here $0.02645 run on subscription; shown at API list price for comparison $26.45
Claude Fable 5.1 100.0% the best score here $0.02928 run on subscription; shown at API list price for comparison $29.28

Good enough pick. Claude Haiku 4.5 at 100.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $2.01 for 1,000 tasks a month.

tested in Claude Code, Opus 5.5 series, each task asked once

Instructions before or after, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 100.0% 100.0% to 100.0% 10 items 100.0% C02-instruction-after $0.00332 run on subscription; shown at API list price for comparison 713 in, 23 out 16 one pass, not measured asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here 11.7 s 11.6 s to 12.1 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Instructions before or after: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 100.0% the best score here $0.00332 run on subscription; shown at API list price for comparison $3.32

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Instructions before or after, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 100.0% interval not measured 10 items 100.0% C02-instruction-after not measured 4,010 in, 9 out 0 one pass, not measured not measured
1GPT-5.6 Luna 100.0% interval not measured 10 items 100.0% C02-instruction-after not measured 4,754 in, 9 out 0 one pass, not measured not measured
1GPT-5.6 Terra 100.0% interval not measured 10 items 100.0% C02-instruction-after not measured 6,248 in, 9 out 0 one pass, not measured not measured
1GPT-6 Luna 100.0% interval not measured 10 items 100.0% C02-instruction-after not measured 6,882 in, 9 out 0 one pass, not measured 7.2 s 6.8 s to 8.0 s across the middle half of 20 calls
1GPT-6 Sol 100.0% interval not measured 10 items 100.0% C02-instruction-after not measured 6,812 in, 9 out 0 one pass, not measured 7.8 s 7.0 s to 8.9 s across the middle half of 20 calls

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Instructions before or after: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 100.0% the best score here not measured not measured
GPT-5.6 Luna 100.0% the best score here not measured not measured
GPT-5.6 Terra 100.0% the best score here not measured not measured
GPT-6 Luna 100.0% the best score here not measured not measured
GPT-6 Sol 100.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Instructions in long prompts

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Instructions in long prompts, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 100.0% 100.0% to 100.0% 50 records 100.0% C07-instruction-at-end not measured 1,386 in, 878 out not measured 20 of 20 every repeat agreed at temperature 0 not measured
1Gemini 3.1 Flash Lite 100.0% 100.0% to 100.0% 50 records 100.0% C07-instruction-at-end not measured 1,249 in, 398 out not measured 20 of 20 every repeat agreed at temperature 0 asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
1GPT-5 mini 100.0% 100.0% to 100.0% 50 records 100.0% C07-instruction-at-end not measured 1,274 in, 845 out not measured 20 of 20 every repeat agreed at the vendor default asked five times: 10 of 10 the same every time, 72 to 100 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 50 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Instructions in long prompts: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 100.0% the best score here not measured not measured
Gemini 3.1 Flash Lite 100.0% the best score here not measured not measured
GPT-5 mini 100.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code

Instructions in long prompts, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 100.0% 100.0% to 100.0% 33 records 100.0% C07-instruction-at-end $0.23998 run on subscription; shown at API list price for comparison 4,267 in, 3,946 out 783 17 of 17 every repeat agreed no sampling control here asked five times: withheld: 8 of 10 tasks got five usable answers not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
1Claude Fable 5.1 100.0% 100.0% to 100.0% 50 records 100.0% C07-instruction-at-end $0.16619 run on subscription; shown at API list price for comparison 6,464 in, 2,031 out 362 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
1Claude Haiku 4.5 100.0% 100.0% to 100.0% 50 records 100.0% C07-instruction-at-end $0.00664 run on subscription; shown at API list price for comparison 2,211 in, 885 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
1Claude Opus 5 100.0% 100.0% to 100.0% 50 records 100.0% C07-instruction-at-end $0.08287 run on subscription; shown at API list price for comparison 3,170 in, 2,681 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
1Claude Sonnet 5 100.0% 100.0% to 100.0% 50 records 100.0% C07-instruction-at-end $0.02143 run on subscription; shown at API list price for comparison 4,800 in, 1,183 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Instructions in long prompts: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 100.0% the best score here $0.00664 run on subscription; shown at API list price for comparison $6.64
Claude Sonnet 5 100.0% the best score here $0.02143 run on subscription; shown at API list price for comparison $21.43
Claude Opus 5 100.0% the best score here $0.08287 run on subscription; shown at API list price for comparison $82.87
Claude Fable 5.1 100.0% the best score here $0.16619 run on subscription; shown at API list price for comparison $166.19
Claude Fable 5 100.0% the best score here $0.23998 run on subscription; shown at API list price for comparison $239.98

Good enough pick. Claude Haiku 4.5 at 100.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $6.64 for 1,000 tasks a month.

tested in Claude Code, Opus 5.5 series, each task asked once

Instructions in long prompts, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 100.0% 100.0% to 100.0% 10 items 100.0% C07-instruction-at-end $0.01170 run on subscription; shown at API list price for comparison 854 in, 414 out 107 one pass, not measured asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here 15.0 s 14.4 s to 16.2 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Instructions in long prompts: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 100.0% the best score here $0.01170 run on subscription; shown at API list price for comparison $11.70

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Instructions in long prompts, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 100.0% interval not measured 10 items 100.0% C07-instruction-at-end not measured 4,086 in, 162 out 0 one pass, not measured not measured
1GPT-5.6 Luna 100.0% interval not measured 10 items 100.0% C07-instruction-at-end not measured 4,829 in, 94 out 0 one pass, not measured not measured
1GPT-5.6 Terra 100.0% interval not measured 10 items 100.0% C07-instruction-at-end not measured 6,343 in, 100 out 0 one pass, not measured not measured
1GPT-6 Luna 100.0% interval not measured 10 items 100.0% C07-instruction-at-end not measured 6,887 in, 109 out 0 one pass, not measured 9.8 s 9.0 s to 10.8 s across the middle half of 20 calls
1GPT-6 Sol 100.0% interval not measured 10 items 100.0% C07-instruction-at-end not measured 6,887 in, 103 out 0 one pass, not measured 10.2 s 9.7 s to 10.7 s across the middle half of 20 calls

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Instructions in long prompts: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 100.0% the best score here not measured not measured
GPT-5.6 Luna 100.0% the best score here not measured not measured
GPT-5.6 Terra 100.0% the best score here not measured not measured
GPT-6 Luna 100.0% the best score here not measured not measured
GPT-6 Sol 100.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Offering the model money

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Offering the model money, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelAnswer length, on its ownBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Gemini 3.1 Flash Lite 519 words 506 to 531 50 answers 575 words C08-tip-length not measured 59 in, 3,513 out not measured 20 of 20 every repeat agreed at temperature 0 asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
2GPT-5 mini 355 words 334 to 375 50 answers 534 words C08-tip-length not measured 90 in, 2,337 out not measured 0 of 20 71 words apart where not at the vendor default asked five times: 0 of 10 the same every time, 0 to 28 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip
3Claude Haiku 4.5 204 words 199 to 209 50 answers 248 words C08-tip-length not measured 103 in, 1,619 out not measured 3 of 20 17 words apart where not at temperature 0 not measured

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 50 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Offering the model money: cost for 1,000 unaided task attempts a month.
ModelAnswer length, on its ownCost per taskAt this volume
Claude Haiku 4.5 204 words interval does not reach the best not measured not measured
Gemini 3.1 Flash Lite 519 words the best score here not measured not measured
GPT-5 mini 355 words interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code

Offering the model money, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelAnswer length, on its ownBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5 544 words 509 to 579 50 answers 828 words C08-tip-length $0.15452 run on subscription; shown at API list price for comparison 1,247 in, 5,931 out 0 a gate holding at zero, not a measured zero 0 of 20 112 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
2Claude Fable 5.1 403 words 385 to 421 50 answers 508 words C08-tip-length $0.23338 run on subscription; shown at API list price for comparison 1,257 in, 4,416 out 3 0 of 20 58 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
3Claude Sonnet 5 377 words 362 to 392 50 answers 457 words C08-tip-length $0.04774 run on subscription; shown at API list price for comparison 2,877 in, 4,199 out 0 a gate holding at zero, not a measured zero 0 of 20 44 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip
4Claude Fable 5 325 words 313 to 336 50 answers 387 words C08-tip-length $0.20286 run on subscription; shown at API list price for comparison 1,247 in, 3,808 out 86 0 of 20 32 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
5Claude Haiku 4.5 241 words 236 to 247 50 answers 258 words C08-tip-length $0.01031 run on subscription; shown at API list price for comparison 928 in, 1,877 out 0 a gate holding at zero, not a measured zero 0 of 20 12 words apart where not no sampling control here asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Offering the model money: cost for 1,000 unaided task attempts a month.
ModelAnswer length, on its ownCost per taskAt this volume
Claude Haiku 4.5 241 words interval does not reach the best $0.01031 run on subscription; shown at API list price for comparison $10.31
Claude Sonnet 5 377 words interval does not reach the best $0.04774 run on subscription; shown at API list price for comparison $47.75
Claude Opus 5 544 words the best score here $0.15452 run on subscription; shown at API list price for comparison $154.52
Claude Fable 5 325 words interval does not reach the best $0.20286 run on subscription; shown at API list price for comparison $202.86
Claude Fable 5.1 403 words interval does not reach the best $0.23338 run on subscription; shown at API list price for comparison $233.38

Good enough pick. Claude Opus 5 at 544 words is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $154.52 for 1,000 tasks a month.

tested in Claude Code, Opus 5.5 series, each task asked once

Offering the model money, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelAnswer length, on its ownBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 508 words 466 to 549 10 items 633 words C08-tip-length $0.02539 run on subscription; shown at API list price for comparison 469 in, 1,176 out 31 one pass, not measured asked five times: 0 of 10 the same every time, 0 to 28 percent, no sampling control here 23.9 s 23.4 s to 25.7 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Offering the model money: cost for 1,000 unaided task attempts a month.
ModelAnswer length, on its ownCost per taskAt this volume
Claude Opus 5.5 508 words the best score here $0.02539 run on subscription; shown at API list price for comparison $25.39

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Offering the model money, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelAnswer length, on its ownBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 193 words interval not measured 10 items 296 words C08-tip-length not measured 3,850 in, 260 out 0 one pass, not measured not measured
2GPT-5.6 Luna 109 words interval not measured 10 items 192 words C08-tip-length not measured 4,592 in, 148 out 0 one pass, not measured not measured
3GPT-5.6 Terra 103 words interval not measured 10 items 185 words C08-tip-length not measured 6,106 in, 138 out 0 one pass, not measured not measured
4GPT-6 Luna 89 words interval not measured 10 items 127 words C08-tip-length not measured 6,718 in, 116 out 0 one pass, not measured 11.4 s 10.5 s to 14.0 s across the middle half of 20 calls
5GPT-6 Sol 79 words interval not measured 10 items 127 words C08-tip-length not measured 6,650 in, 105 out 0 one pass, not measured 10.9 s 10.2 s to 12.2 s across the middle half of 20 calls

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Offering the model money: cost for 1,000 unaided task attempts a month.
ModelAnswer length, on its ownCost per taskAt this volume
GPT-5.4 mini 193 words the best score here not measured not measured
GPT-5.6 Luna 109 words interval does not reach the best not measured not measured
GPT-5.6 Terra 103 words interval does not reach the best not measured not measured
GPT-6 Luna 89 words interval does not reach the best not measured not measured
GPT-6 Sol 79 words interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Questions about recent events

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Questions about recent events, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 85.6% 73.2% to 98.0% 30 items 62.2% C16-cutoff-disclosure not measured 58 in, 162 out not measured 28 of 30 66.7 percentage points apart where not 56 more answered once temperature 0 asked for not measured
2GPT-5 mini 48.6% 31.8% to 65.4% 30 items 89.7% C16-cutoff-disclosure not measured 52 in, 131 out not measured 21 of 30 61.1 percentage points apart where not 56 more answered once temperature 0 asked for asked five times: 6 of 30 the same every time, 10 to 37 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip
3Gemini 3.1 Flash Lite 3.3% 0.0% to 9.9%† 30 items 33.3% C16-cutoff-disclosure not measured 44 in, 90 out not measured 30 of 30 every repeat agreed 56 more answered once temperature 0 asked for asked five times: 30 of 30 the same every time, 89 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

0 of the 108 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Questions about recent events: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 85.6% the best score here not measured not measured
Gemini 3.1 Flash Lite 3.3% interval does not reach the best not measured not measured
GPT-5 mini 48.6% interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested via API, API twin for the Codex run

Questions about recent events, tested via API, API twin for the Codex run. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 27.5% interval not measured 30 items 86.7% C16-cutoff-disclosure $0.00048 billed through the API 52 in, 97 out not measured 25 of 30 63.3 percentage points apart where not 56 more answered once at temperature 0 not measured

0 of the 108 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Questions about recent events: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 27.5% the best score here $0.00048 billed through the API $0.48

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Claude Code

Questions about recent events, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 96.7% 90.1% to 100.0%† 30 items 99.2% C16-cutoff-disclosure $0.00220 run on subscription; shown at API list price for comparison 474 in, 345 out 0 a gate holding at zero, not a measured zero 19 of 20 50.0 percentage points apart where not 40 more answered once no sampling control here asked five times: 28 of 30 the same every time, 79 to 98 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
2Claude Sonnet 5 93.3% 84.3% to 100.0%† 30 items 92.5% C16-cutoff-disclosure $0.00757 run on subscription; shown at API list price for comparison 1,448 in, 467 out 0 a gate holding at zero, not a measured zero 19 of 20 50.0 percentage points apart where not 40 more answered once no sampling control here asked five times: 22 of 30 the same every time, 56 to 86 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip
3Claude Fable 5.1 92.2% 83.0% to 100.0%† 30 items 89.2% C16-cutoff-disclosure $0.04170 run on subscription; shown at API list price for comparison 638 in, 706 out 347 18 of 20 58.3 percentage points apart where not 40 more answered once no sampling control here asked five times: 20 of 30 the same every time, 49 to 81 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
4Claude Fable 5 87.8% 77.5% to 98.0% 30 items 95.6% C16-cutoff-disclosure $0.04206 run on subscription; shown at API list price for comparison 633 in, 715 out 336 15 of 20 60.0 percentage points apart where not 40 more answered once no sampling control here asked five times: 20 of 30 the same every time, 49 to 81 percent, no sampling control here not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
5Claude Opus 5 80.3% 67.4% to 93.1% 30 items 86.7% C16-cutoff-disclosure $0.01566 run on subscription; shown at API list price for comparison 633 in, 500 out 0 a gate holding at zero, not a measured zero 13 of 20 59.5 percentage points apart where not 40 more answered once no sampling control here asked five times: 23 of 30 the same every time, 59 to 88 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Questions about recent events: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 96.7% the best score here $0.00220 run on subscription; shown at API list price for comparison $2.20
Claude Sonnet 5 93.3% interval overlaps the best $0.00757 run on subscription; shown at API list price for comparison $7.57
Claude Opus 5 80.3% interval overlaps the best $0.01566 run on subscription; shown at API list price for comparison $15.66
Claude Fable 5.1 92.2% interval overlaps the best $0.04170 run on subscription; shown at API list price for comparison $41.70
Claude Fable 5 87.8% interval overlaps the best $0.04206 run on subscription; shown at API list price for comparison $42.06

Good enough pick. Claude Haiku 4.5 at 96.7% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $2.20 for 1,000 tasks a month.

tested in Claude Code, Opus 5.5 series, each task asked once

Questions about recent events, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 100.0% 100.0% to 100.0% 30 items 93.3% C16-cutoff-disclosure $0.00756 run on subscription; shown at API list price for comparison 474 in, 283 out 161 one pass, not measured asked five times: 24 of 30 the same every time, 63 to 90 percent, no sampling control here 14.1 s 13.7 s to 14.5 s across the middle half of 60 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Questions about recent events: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 100.0% the best score here $0.00756 run on subscription; shown at API list price for comparison $7.56

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, replaying two earlier tests

Questions about recent events, tested in Codex, Codex run, replaying two earlier tests. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 15.0% interval not measured 30 items 69.2% C16-cutoff-disclosure not measured 19,960 in, 107 out 0 19 of 30 62.1 percentage points apart where not 56 more answered once setting not recorded not measured

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Questions about recent events: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 15.0% the best score here not measured not measured

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Questions about recent events, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.6 Terra 30.0% interval not measured 30 items 40.0% C16-cutoff-disclosure not measured 6,516 in, 27 out 0 one pass, not measured not measured

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Questions about recent events: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.6 Terra 30.0% the best score here not measured not measured

No good enough pick. One model ran this job this way, so there is nothing to pick between.

Showing examples

Showing examples

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown
tested in Claude Code
Showing examples, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 100.0% 100.0% to 100.0% 50 records 100.0% C04-three-shot-format $0.03131 run on subscription; shown at API list price for comparison 1,719 in, 283 out 132 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
1Claude Fable 5.1 100.0% 100.0% to 100.0% 50 records 100.0% C04-three-shot-format $0.02474 run on subscription; shown at API list price for comparison 1,729 in, 149 out 0 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
1Claude Haiku 4.5 100.0% 100.0% to 100.0% 50 records 100.0% C04-three-shot-format $0.00187 run on subscription; shown at API list price for comparison 1,281 in, 118 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
1Claude Opus 5 100.0% 100.0% to 100.0% 50 records 100.0% C04-three-shot-format $0.01238 run on subscription; shown at API list price for comparison 1,719 in, 151 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
1Claude Sonnet 5 100.0% 100.0% to 100.0% 50 records 100.0% C04-three-shot-format $0.00822 run on subscription; shown at API list price for comparison 3,349 in, 152 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 100.0% the best score here $0.00187 run on subscription; shown at API list price for comparison $1.87
Claude Sonnet 5 100.0% the best score here $0.00822 run on subscription; shown at API list price for comparison $8.22
Claude Opus 5 100.0% the best score here $0.01238 run on subscription; shown at API list price for comparison $12.38
Claude Fable 5.1 100.0% the best score here $0.02474 run on subscription; shown at API list price for comparison $24.74
Claude Fable 5 100.0% the best score here $0.03131 run on subscription; shown at API list price for comparison $31.31

Good enough pick. Claude Haiku 4.5 at 100.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $1.87 for 1,000 tasks a month.

tested in Claude Code, Opus 5.5 series, each task asked once
Showing examples, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 100.0% 100.0% to 100.0% 10 items 100.0% C04-three-shot-format $0.00284 run on subscription; shown at API list price for comparison 564 in, 29 out 0 one pass, not measured asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here 11.9 s 11.8 s to 12.1 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 100.0% the best score here $0.00284 run on subscription; shown at API list price for comparison $2.84

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip
Showing examples, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 100.0% interval not measured 10 items 100.0% C04-three-shot-format not measured 3,894 in, 22 out 0 one pass, not measured not measured
1GPT-5.6 Luna 100.0% interval not measured 10 items 100.0% C04-three-shot-format not measured 4,637 in, 22 out 0 one pass, not measured not measured
1GPT-5.6 Terra 100.0% interval not measured 10 items 100.0% C04-three-shot-format not measured 6,173 in, 22 out 0 one pass, not measured not measured
1GPT-6 Luna 100.0% interval not measured 10 items 100.0% C04-three-shot-format not measured 6,719 in, 22 out 0 one pass, not measured 7.3 s 7.1 s to 8.0 s across the middle half of 20 calls
1GPT-6 Sol 100.0% interval not measured 10 items 100.0% C04-three-shot-format not measured 6,718 in, 22 out 0 one pass, not measured 8.0 s 7.6 s to 8.2 s across the middle half of 20 calls

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 100.0% the best score here not measured not measured
GPT-5.6 Luna 100.0% the best score here not measured not measured
GPT-5.6 Terra 100.0% the best score here not measured not measured
GPT-6 Luna 100.0% the best score here not measured not measured
GPT-6 Sol 100.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Showing examples

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown
tested via API, Main run, tip by tip
Showing examples, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 81.4% 77.2% to 85.7% 10 items 97.1% C04b-zero-vs-one not measured 971 in, 221 out not measured 20 of 20 every repeat agreed temperature 0 asked for not measured
2Gemini 3.1 Flash Lite 80.0% 75.4% to 84.6% 10 items 95.7% C04b-zero-vs-one not measured 871 in, 199 out not measured 20 of 20 every repeat agreed temperature 0 asked for asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
3GPT-5 mini 78.3% 73.0% to 83.6% 10 items 96.9% C04b-zero-vs-one not measured 875 in, 218 out not measured 17 of 20 7.6 percentage points apart where not temperature 0 asked for asked five times: 9 of 10 the same every time, 60 to 98 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip

0 of the 60 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 81.4% the best score here not measured not measured
Gemini 3.1 Flash Lite 80.0% interval overlaps the best not measured not measured
GPT-5 mini 78.3% interval overlaps the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code
Showing examples, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 82.3% 78.6% to 86.0% 10 items 99.7% C04b-zero-vs-one $0.08170 run on subscription; shown at API list price for comparison 2,621 in, 1,110 out 872 18 of 20 7.1 percentage points apart where not no sampling control here asked five times: 8 of 10 the same every time, 49 to 94 percent, no sampling control here not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
2Claude Fable 5.1 81.7% 77.7% to 85.7% 10 items 100.0% C04b-zero-vs-one $0.05013 run on subscription; shown at API list price for comparison 2,590 in, 485 out 252 19 of 20 5.7 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
3Claude Haiku 4.5 81.4% 77.2% to 85.7% 10 items 94.3% C04b-zero-vs-one $0.00311 run on subscription; shown at API list price for comparison 1,961 in, 230 out 0 a gate holding at zero, not a measured zero 18 of 20 5.7 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
3Claude Opus 5 81.4% 77.2% to 85.7% 10 items 98.0% C04b-zero-vs-one $0.01882 run on subscription; shown at API list price for comparison 2,578 in, 237 out 0 a gate holding at zero, not a measured zero 18 of 20 7.1 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
3Claude Sonnet 5 81.4% 77.2% to 85.7% 10 items 97.7% C04b-zero-vs-one $0.01142 run on subscription; shown at API list price for comparison 4,534 in, 235 out 0 a gate holding at zero, not a measured zero 18 of 20 7.1 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 81.4% interval overlaps the best $0.00311 run on subscription; shown at API list price for comparison $3.11
Claude Sonnet 5 81.4% interval overlaps the best $0.01142 run on subscription; shown at API list price for comparison $11.42
Claude Opus 5 81.4% interval overlaps the best $0.01882 run on subscription; shown at API list price for comparison $18.82
Claude Fable 5.1 81.7% interval overlaps the best $0.05013 run on subscription; shown at API list price for comparison $50.13
Claude Fable 5 82.3% the best score here $0.08170 run on subscription; shown at API list price for comparison $81.70

Good enough pick. Claude Haiku 4.5 at 81.4% and $3.11 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Fable 5, the best on this table at 82.3% and $81.70.

tested in Claude Code, Opus 5.5 series, each task asked once
Showing examples, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 82.9% 79.1% to 86.6% 10 items 100.0% C04b-zero-vs-one $0.00553 run on subscription; shown at API list price for comparison 650 in, 147 out 108 one pass, not measured asked five times: 9 of 10 the same every time, 60 to 98 percent, no sampling control here 12.8 s 12.4 s to 13.1 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 82.9% the best score here $0.00553 run on subscription; shown at API list price for comparison $5.53

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip
Showing examples, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.6 Luna 82.9% interval not measured 10 items 95.7% C04b-zero-vs-one not measured 4,720 in, 32 out 0 one pass, not measured not measured
2GPT-5.6 Terra 81.4% interval not measured 10 items 100.0% C04b-zero-vs-one not measured 6,214 in, 31 out 0 one pass, not measured not measured
2GPT-6 Luna 81.4% interval not measured 10 items 95.7% C04b-zero-vs-one not measured 6,778 in, 31 out 0 one pass, not measured 7.6 s 7.3 s to 8.2 s across the middle half of 20 calls
2GPT-6 Sol 81.4% interval not measured 10 items 98.6% C04b-zero-vs-one not measured 6,847 in, 31 out 0 one pass, not measured 8.0 s 7.7 s to 8.7 s across the middle half of 20 calls
3GPT-5.4 mini 80.0% interval not measured 10 items 100.0% C04b-zero-vs-one not measured 3,956 in, 31 out 0 one pass, not measured not measured

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 80.0% interval does not reach the best not measured not measured
GPT-5.6 Luna 82.9% the best score here not measured not measured
GPT-5.6 Terra 81.4% interval does not reach the best not measured not measured
GPT-6 Luna 81.4% interval does not reach the best not measured not measured
GPT-6 Sol 81.4% interval does not reach the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Showing examples

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown
tested via API, Main run, tip by tip
Showing examples, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 81.4% 77.2% to 85.7% 10 items 100.0% C04b-zero-vs-three not measured 971 in, 221 out not measured 20 of 20 every repeat agreed temperature 0 asked for not measured
2Gemini 3.1 Flash Lite 80.0% 75.4% to 84.6% 10 items 98.6% C04b-zero-vs-three not measured 871 in, 199 out not measured 20 of 20 every repeat agreed temperature 0 asked for asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
3GPT-5 mini 79.4% 74.3% to 84.6% 10 items 97.7% C04b-zero-vs-three not measured 875 in, 219 out not measured 18 of 20 8.6 percentage points apart where not temperature 0 asked for asked five times: 9 of 10 the same every time, 60 to 98 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip

0 of the 60 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 81.4% the best score here not measured not measured
Gemini 3.1 Flash Lite 80.0% interval overlaps the best not measured not measured
GPT-5 mini 79.4% interval overlaps the best not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code
Showing examples, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 82.0% 78.2% to 85.8% 10 items 100.0% C04b-zero-vs-three $0.07734 run on subscription; shown at API list price for comparison 2,578 in, 1,031 out 798 19 of 20 8.6 percentage points apart where not no sampling control here asked five times: withheld: 9 of 10 tasks got five usable answers not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
2Claude Sonnet 5 81.7% 77.7% to 85.7% 10 items 100.0% C04b-zero-vs-three $0.01142 run on subscription; shown at API list price for comparison 4,534 in, 236 out 0 a gate holding at zero, not a measured zero 19 of 20 5.7 percentage points apart where not no sampling control here asked five times: 9 of 10 the same every time, 60 to 98 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip
3Claude Fable 5.1 81.4% 77.2% to 85.7% 10 items 100.0% C04b-zero-vs-three $0.05010 run on subscription; shown at API list price for comparison 2,590 in, 484 out 252 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
3Claude Haiku 4.5 81.4% 77.2% to 85.7% 10 items 98.9% C04b-zero-vs-three $0.00310 run on subscription; shown at API list price for comparison 1,961 in, 228 out 0 a gate holding at zero, not a measured zero 19 of 20 5.7 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
3Claude Opus 5 81.4% 77.2% to 85.7% 10 items 99.7% C04b-zero-vs-three $0.01882 run on subscription; shown at API list price for comparison 2,578 in, 237 out 0 a gate holding at zero, not a measured zero 19 of 20 5.7 percentage points apart where not no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 81.4% interval overlaps the best $0.00310 run on subscription; shown at API list price for comparison $3.10
Claude Sonnet 5 81.7% interval overlaps the best $0.01142 run on subscription; shown at API list price for comparison $11.42
Claude Opus 5 81.4% interval overlaps the best $0.01882 run on subscription; shown at API list price for comparison $18.82
Claude Fable 5.1 81.4% interval overlaps the best $0.05010 run on subscription; shown at API list price for comparison $50.10
Claude Fable 5 82.0% the best score here $0.07734 run on subscription; shown at API list price for comparison $77.34

Good enough pick. Claude Haiku 4.5 at 81.4% and $3.10 for 1,000 tasks a month is the cheapest model whose bracket overlaps Claude Fable 5, the best on this table at 82.0% and $77.34.

tested in Claude Code, Opus 5.5 series, each task asked once
Showing examples, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 82.9% 79.1% to 86.6% 10 items 100.0% C04b-zero-vs-three $0.00557 run on subscription; shown at API list price for comparison 650 in, 148 out 110 one pass, not measured asked five times: 9 of 10 the same every time, 60 to 98 percent, no sampling control here 13.1 s 12.7 s to 13.3 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 82.9% the best score here $0.00557 run on subscription; shown at API list price for comparison $5.57

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip
Showing examples, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.6 Luna 81.4% interval not measured 10 items 100.0% C04b-zero-vs-three not measured 4,719 in, 33 out 0 one pass, not measured not measured
1GPT-5.6 Terra 81.4% interval not measured 10 items 100.0% C04b-zero-vs-three not measured 6,215 in, 31 out 0 one pass, not measured not measured
1GPT-6 Luna 81.4% interval not measured 10 items 100.0% C04b-zero-vs-three not measured 6,778 in, 32 out 0 one pass, not measured 7.3 s 7.2 s to 7.5 s across the middle half of 20 calls
1GPT-6 Sol 81.4% interval not measured 10 items 100.0% C04b-zero-vs-three not measured 6,779 in, 31 out 0 one pass, not measured 8.0 s 7.7 s to 8.5 s across the middle half of 20 calls
2GPT-5.4 mini 78.6% interval not measured 10 items 100.0% C04b-zero-vs-three not measured 3,956 in, 31 out 0 one pass, not measured not measured

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Showing examples: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 78.6% interval does not reach the best not measured not measured
GPT-5.6 Luna 81.4% the best score here not measured not measured
GPT-5.6 Terra 81.4% the best score here not measured not measured
GPT-6 Luna 81.4% the best score here not measured not measured
GPT-6 Sol 81.4% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Stop made-up answers

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Stop made-up answers, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5 mini 20.0% 8.8% to 31.2% 50 records 100.0% C06-permit-idk not measured 231 in, 165 out not measured 15 of 20 56.0 percentage points apart where not at the vendor default asked five times: 3 of 10 the same every time, 11 to 60 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip
2Claude Haiku 4.5 0.0% 0.0% to 0.0% 50 records 100.0% C06-permit-idk not measured 256 in, 271 out not measured 20 of 20 every repeat agreed at temperature 0 not measured
2Gemini 3.1 Flash Lite 0.0% 0.0% to 0.0% 50 records 100.0% C06-permit-idk not measured 222 in, 87 out not measured 20 of 20 every repeat agreed at temperature 0 asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 50 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Stop made-up answers: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 0.0% interval does not reach the best not measured not measured
Gemini 3.1 Flash Lite 0.0% interval does not reach the best not measured not measured
GPT-5 mini 20.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code

Stop made-up answers, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5.1 76.0% 64.0% to 88.0% 50 records 100.0% C06-permit-idk $0.03175 run on subscription; shown at API list price for comparison 1,450 in, 345 out 0 17 of 20 46.7 percentage points apart where not no sampling control here asked five times: 6 of 10 the same every time, 31 to 83 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
2Claude Sonnet 5 38.0% 24.4% to 51.6% 50 records 100.0% C06-permit-idk $0.01083 run on subscription; shown at API list price for comparison 3,070 in, 469 out 0 a gate holding at zero, not a measured zero 16 of 20 50.0 percentage points apart where not no sampling control here asked five times: 5 of 10 the same every time, 24 to 76 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip
3Claude Opus 5 30.0% 17.2% to 42.8% 50 records 100.0% C06-permit-idk $0.02363 run on subscription; shown at API list price for comparison 1,440 in, 657 out 0 a gate holding at zero, not a measured zero 15 of 20 40.0 percentage points apart where not no sampling control here asked five times: 8 of 10 the same every time, 49 to 94 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
4Claude Fable 5 24.0% 12.0% to 36.0% 50 records 100.0% C06-permit-idk $0.03501 run on subscription; shown at API list price for comparison 1,440 in, 412 out 44 14 of 20 46.7 percentage points apart where not no sampling control here asked five times: withheld: 9 of 10 tasks got five usable answers not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
5Claude Haiku 4.5 22.0% 10.4% to 33.6% 50 records 100.0% C06-permit-idk $0.00340 run on subscription; shown at API list price for comparison 1,081 in, 463 out 0 a gate holding at zero, not a measured zero 16 of 20 60.0 percentage points apart where not no sampling control here asked five times: 9 of 10 the same every time, 60 to 98 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Stop made-up answers: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 22.0% interval does not reach the best $0.00340 run on subscription; shown at API list price for comparison $3.40
Claude Sonnet 5 38.0% interval does not reach the best $0.01083 run on subscription; shown at API list price for comparison $10.83
Claude Opus 5 30.0% interval does not reach the best $0.02363 run on subscription; shown at API list price for comparison $23.63
Claude Fable 5.1 76.0% the best score here $0.03175 run on subscription; shown at API list price for comparison $31.75
Claude Fable 5 24.0% interval does not reach the best $0.03501 run on subscription; shown at API list price for comparison $35.01

Good enough pick. Claude Fable 5.1 at 76.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $31.75 for 1,000 tasks a month.

tested in Claude Code, Opus 5.5 series, each task asked once

Stop made-up answers, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 90.0% 70.4% to 100.0%† 10 items 100.0% C06-permit-idk $0.00382 run on subscription; shown at API list price for comparison 508 in, 89 out 13 one pass, not measured asked five times: 6 of 10 the same every time, 31 to 83 percent, no sampling control here 12.2 s 12.0 s to 13.0 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

† The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Stop made-up answers: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 90.0% the best score here $0.00382 run on subscription; shown at API list price for comparison $3.82

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Stop made-up answers, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 30.0% interval not measured 10 items 90.0% C06-permit-idk not measured 3,857 in, 25 out 0 one pass, not measured not measured
1GPT-5.6 Luna 30.0% interval not measured 10 items 100.0% C06-permit-idk not measured 4,599 in, 17 out 0 one pass, not measured not measured
1GPT-6 Sol 30.0% interval not measured 10 items 100.0% C06-permit-idk not measured 6,678 in, 19 out 0 one pass, not measured 8.2 s 7.7 s to 8.5 s across the middle half of 20 calls
2GPT-5.6 Terra 20.0% interval not measured 10 items 100.0% C06-permit-idk not measured 6,134 in, 15 out 0 one pass, not measured not measured
3GPT-6 Luna 0.0% interval not measured 10 items 100.0% C06-permit-idk not measured 6,679 in, 21 out 0 one pass, not measured 7.7 s 7.1 s to 8.5 s across the middle half of 20 calls

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Stop made-up answers: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 30.0% the best score here not measured not measured
GPT-5.6 Luna 30.0% the best score here not measured not measured
GPT-5.6 Terra 20.0% interval does not reach the best not measured not measured
GPT-6 Luna 0.0% interval does not reach the best not measured not measured
GPT-6 Sol 30.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Using tags and formatting

Every model that ran this job is below, with the score it got on its own and what each task cost. Each test method has a table of its own, and the two are never added together.

Show the full breakdown

tested via API, Main run, tip by tip

Using tags and formatting, tested via API, Main run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 100.0% 100.0% to 100.0% 50 records 100.0% C03-xml-delimiters not measured 749 in, 759 out not measured 20 of 20 every repeat agreed at temperature 0 not measured
1Gemini 3.1 Flash Lite 100.0% 100.0% to 100.0% 50 records 100.0% C03-xml-delimiters not measured 686 in, 368 out not measured 20 of 20 every repeat agreed at temperature 0 asked five times: 10 of 10 the same every time, 72 to 100 percent, at temperature 0 not measured asked five times: 563 ms, middle half 460 ms to 751 ms over 700 calls on every tip
1GPT-5 mini 100.0% 100.0% to 100.0% 50 records 100.0% C03-xml-delimiters not measured 708 in, 771 out not measured 20 of 20 every repeat agreed at the vendor default asked five times: 10 of 10 the same every time, 72 to 100 percent, at the vendor default not measured asked five times: 1.5 s, middle half 1.1 s to 2.4 s over 702 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

0 of the 50 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed on this run for the version being priced here.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Using tags and formatting: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 100.0% the best score here not measured not measured
Gemini 3.1 Flash Lite 100.0% the best score here not measured not measured
GPT-5 mini 100.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

tested in Claude Code

Using tags and formatting, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Fable 5 100.0% 100.0% to 100.0% 48 records 100.0% C03-xml-delimiters $0.11688 run on subscription; shown at API list price for comparison 2,370 in, 1,864 out 543 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 15.0 s, middle half 13.2 s to 18.7 s over 729 calls on every tip
1Claude Fable 5.1 100.0% 100.0% to 100.0% 50 records 100.0% C03-xml-delimiters $0.17308 run on subscription; shown at API list price for comparison 2,193 in, 3,023 out 1,351 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 13.0 s, middle half 11.3 s to 16.2 s over 700 calls on every tip
1Claude Haiku 4.5 100.0% 100.0% to 100.0% 50 records 100.0% C03-xml-delimiters $0.00533 run on subscription; shown at API list price for comparison 1,574 in, 752 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.4 s, middle half 10.5 s to 12.2 s over 700 calls on every tip
1Claude Opus 5 100.0% 100.0% to 100.0% 50 records 100.0% C03-xml-delimiters $0.04752 run on subscription; shown at API list price for comparison 2,183 in, 1,464 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 12.0 s, middle half 11.0 s to 14.6 s over 700 calls on every tip
1Claude Sonnet 5 100.0% 100.0% to 100.0% 50 records 100.0% C03-xml-delimiters $0.01651 run on subscription; shown at API list price for comparison 3,813 in, 889 out 0 a gate holding at zero, not a measured zero 20 of 20 every repeat agreed no sampling control here asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here not measured asked five times: 11.8 s, middle half 10.8 s to 13.0 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Using tags and formatting: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 100.0% the best score here $0.00533 run on subscription; shown at API list price for comparison $5.33
Claude Sonnet 5 100.0% the best score here $0.01651 run on subscription; shown at API list price for comparison $16.51
Claude Opus 5 100.0% the best score here $0.04752 run on subscription; shown at API list price for comparison $47.52
Claude Fable 5 100.0% the best score here $0.11688 run on subscription; shown at API list price for comparison $116.88
Claude Fable 5.1 100.0% the best score here $0.17308 run on subscription; shown at API list price for comparison $173.08

Good enough pick. Claude Haiku 4.5 at 100.0% is both the best score on this table and the cheapest of the models its bracket cannot separate from it, at $5.33 for 1,000 tasks a month.

tested in Claude Code, Opus 5.5 series, each task asked once

Using tags and formatting, tested in Claude Code, Opus 5.5 series, each task asked once. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Opus 5.5 100.0% 100.0% to 100.0% 10 items 100.0% C03-xml-delimiters $0.01231 run on subscription; shown at API list price for comparison 657 in, 484 out 149 one pass, not measured asked five times: 10 of 10 the same every time, 72 to 100 percent, no sampling control here 16.3 s 15.5 s to 17.2 s across the middle half of 20 calls asked five times: 13.7 s, middle half 12.8 s to 16.1 s over 700 calls on every tip

This model ran at its own default settings. We could not fix them: one vendor will not accept a setting of 0, and Claude Code has no setting to fix. Recorded as a difference, not a failure.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Using tags and formatting: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Opus 5.5 100.0% the best score here $0.01231 run on subscription; shown at API list price for comparison $12.31

No good enough pick. One model ran this job this way, so there is nothing to pick between.

tested in Codex, Codex run, tip by tip

Using tags and formatting, tested in Codex, Codex run, tip by tip. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1GPT-5.4 mini 100.0% interval not measured 10 items 100.0% C03-xml-delimiters not measured 3,973 in, 37 out 0 one pass, not measured not measured
1GPT-5.6 Luna 100.0% interval not measured 10 items 90.0% C03-xml-delimiters not measured 4,715 in, 35 out 0 one pass, not measured not measured
1GPT-5.6 Terra 100.0% interval not measured 10 items 100.0% C03-xml-delimiters not measured 6,230 in, 33 out 0 one pass, not measured not measured
1GPT-6 Luna 100.0% interval not measured 10 items 90.0% C03-xml-delimiters not measured 6,773 in, 35 out 0 one pass, not measured 8.0 s 7.5 s to 8.7 s across the middle half of 20 calls
1GPT-6 Sol 100.0% interval not measured 10 items 100.0% C03-xml-delimiters not measured 6,774 in, 43 out 0 one pass, not measured 8.9 s 8.3 s to 9.5 s across the middle half of 20 calls

Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.

Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Using tags and formatting: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
GPT-5.4 mini 100.0% the best score here not measured not measured
GPT-5.6 Luna 100.0% the best score here not measured not measured
GPT-5.6 Terra 100.0% the best score here not measured not measured
GPT-6 Luna 100.0% the best score here not measured not measured
GPT-6 Sol 100.0% the best score here not measured not measured

No good enough pick. The best model on this table publishes no cost per task, so a cheaper model cannot be priced against it.

Prompting tips, multi-turn

Prompt, then revise

Every model that ran this job is below, with the score it got on its own and what each task cost.

Show the full breakdown

tested in Claude Code

Prompt, then revise, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 95.7% interval not measured 10 items 98.6% C15-iterative-refinement $0.00083 run on subscription; shown at API list price for comparison 557 in, 55 out not measured one pass, not measured not measured

0 of the 10 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Prompt, then revise: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 95.7% the best score here $0.00083 run on subscription; shown at API list price for comparison $0.83

No good enough pick. One model ran this job this way, so there is nothing to pick between.

Rules for the whole chat

Every model that ran this job is below, with the score it got on its own and what each task cost.

Show the full breakdown

tested in Claude Code

Rules for the whole chat, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 100.0% interval not measured 10 items 100.0% C13-system-prompt-placement $0.01660 run on subscription; shown at API list price for comparison 13,384 in, 643 out not measured one pass, not measured not measured

0 of the 10 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Rules for the whole chat: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 100.0% the best score here $0.01660 run on subscription; shown at API list price for comparison $16.60

No good enough pick. One model ran this job this way, so there is nothing to pick between.

Splitting big tasks

Every model that ran this job is below, with the score it got on its own and what each task cost.

Show the full breakdown

tested in Claude Code

Splitting big tasks, tested in Claude Code. Which model does this task best with no help, and what it costs.
RankModelUnaided scoreBest assisted scoreCost per taskTokens per taskThinking per taskSame answer on a repeatTime per call
1Claude Haiku 4.5 68.7% interval not measured 10 items 100.0% C14-prompt-chaining $0.00517 run on subscription; shown at API list price for comparison 3,902 in, 255 out not measured 14 of 20 20.0 percentage points apart where not no sampling control here not measured

0 of the 60 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.

Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.

Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.

What this costs at volume

Cost per task multiplied by the tasks you expect in a month, and nothing else. It is not a bill and nobody was charged it: each figure carries the basis it rests on, and a subscription run is priced at API list price so that it can sit beside a billed one.

Changing this moves every figure below it and cannot move the pick: monthly cost is cost per task times the volume, so no volume reorders two models.

Splitting big tasks: cost for 1,000 unaided task attempts a month.
ModelUnaided scoreCost per taskAt this volume
Claude Haiku 4.5 68.7% the best score here $0.00517 run on subscription; shown at API list price for comparison $5.17

No good enough pick. One model ran this job this way, so there is nothing to pick between.