AI Models Compared on the Same Tasks

Every model we tested, job by job. The full tables are on the comparison page, and the routing page picks a model for each job.

Each job, each model, with no help

Each grid shows how every model scored on each job, with no tip and no skill. There is one grid per test method. The grids are never mixed, because each test method gives the model different tools.

A darker shade is a higher score, and the score is printed too. A solid ring marks the best score in a row. A dashed ring marks a score too close to call against it. Each row reads one run, and the rest are under “Other runs”.

  • 0 to 19%
  • 20 to 39%
  • 40 to 59%
  • 60 to 79%
  • 80 to 100%

Via API

JobClaude Haiku 4.5GPT-5 miniGemini 3.1 Flash LiteGPT-5.4 mini
Accessibility audit Harder tasks, each model at its own tier, 2026-09-18 Different tiers are not ranked against each other.
Other runs (2)
  • First run, with and without the skill, 2026-08-30 GPT-5 mini 69.9% (tier 1), Gemini 3.1 Flash Lite 63.3% (tier 1), Claude Haiku 4.5 40.0% (tier 1)
  • Wider run, three ways, 2026-09-06 Gemini 3.1 Flash Lite 63.3% (tier 1), GPT-5 mini 58.2% (tier 1)
in another run58.7% tier 140.0% tier 2not run
Code review Wider run, three ways, 2026-09-06not run70.0% tier 1 best57.5% tier 1 too close to callnot run
On-page audit First run, with and without the skill, 2026-08-30
Other runs (1)
  • Wider run, three ways, 2026-09-06 GPT-5 mini 52.2% (tier 1), Gemini 3.1 Flash Lite 48.8% (tier 1)
23.9% tier 1 too close to call44.9% tier 1 too close to call48.8% tier 1 bestnot run
Skill authoring First run, with and without the skill, 2026-08-30
Other runs (1)
  • Wider run, three ways, 2026-09-06 Gemini 3.1 Flash Lite 87.3% (tier 1), GPT-5 mini 64.8% (tier 1)
61.8% tier 173.2% tier 1 too close to call87.3% tier 1 bestnot run
Spec writing First run, with and without the skill, 2026-08-30
Other runs (1)
  • Wider run, three ways, 2026-09-06 Gemini 3.1 Flash Lite 99.1% (tier 1), GPT-5 mini 96.4% (tier 1)
99.5% tier 1 best94.5% tier 199.1% tier 1 too close to callnot run
Link graph and metadata parity audit Not run this waynot runnot runnot runnot run
Corpus integrity and correction Not run this waynot runnot runnot runnot run
Traffic drop triage Not run this waynot runnot runnot runnot run
Getting clean JSON back Harder tasks, tier 2, 2026-09-19 No model managed this on its own.
Other runs (1)
  • Main run, tip by tip, 2026-08-17 Claude Haiku 4.5 0.0% (tier 1), Gemini 3.1 Flash Lite 0.0% (tier 1), GPT-5 mini 0.0% (tier 1)
in another run0.0% tier 20.0% tier 2not run
Getting the length right Harder tasks, tier 2, 2026-09-19
Other runs (2)
  • Main run, tip by tip, 2026-08-17 Gemini 3.1 Flash Lite 31.0% (tier 1), Claude Haiku 4.5 24.7% (tier 1), GPT-5 mini 1.6% (tier 1)
  • API twin for the Codex run, 2026-09-07 GPT-5.4 mini 15.7% (tier 1)
in another run5.0% tier 2 best3.7% tier 2 too close to callin another run
Questions about recent events Main run, tip by tip, 2026-08-17 Same tasks, also from: API twin for the Codex run, 2026-09-0785.6% tier 1 best48.6% tier 13.3% tier 127.5% tier 1 same tasks, another run
Better step-by-step answers Main run, tip by tip, 2026-08-1730.0% tier 1 too close to call16.0% tier 140.0% tier 1 bestnot run

In Claude Code

JobClaude Haiku 4.5Claude Fable 5Claude Fable 5.1Claude Opus 5Claude Opus 5.5Claude Sonnet 5
Accessibility audit Harder tasks, each model at its own tier, 2026-09-18 Same tasks, also from: Harder tasks, Claude Opus 5.5 at its own tier, 2026-09-26 Different tiers are not ranked against each other.
Other runs (6)
  • Wider run, three ways, 2026-09-06 Claude Fable 5.1 87.9% (tier 1), Claude Opus 5 87.1% (tier 1), Claude Sonnet 5 80.2% (tier 1), Claude Haiku 4.5 46.1% (tier 1)
  • Second run, every task twice, 2026-08-31 Claude Opus 5 87.3% (tier 1), Claude Sonnet 5 82.3% (tier 1), Claude Haiku 4.5 45.5% (tier 1)
  • Opus 5.5 series, wider run, three ways, 2026-09-26 Claude Opus 5.5 85.8% (tier 1)
  • Fable 5, wider run, three ways, 2026-09-19 Claude Fable 5 89.7% (tier 1)
  • Fable 5.1, single attempt, 2026-09-03 Claude Fable 5.1 85.8% (tier 1)
  • First run, single attempt, 2026-08-31 Claude Opus 5 89.0% (tier 1)
44.8% tier 1in another run89.4% tier 3 too close to call56.1% tier 392.8% tier 3 best same tasks, another run43.3% tier 2
Code review Wider run, three ways, 2026-09-06 Same tasks, also from: Opus 5.5 series, wider run, three ways, 2026-09-26 Same tasks, also from: Fable 5, wider run, three ways, 2026-09-1958.3% tier 185.8% tier 1 too close to call same tasks, another run89.0% tier 1 too close to call93.5% tier 1 best85.4% tier 1 too close to call same tasks, another run74.9% tier 1 too close to call
On-page audit Wider run, three ways, 2026-09-06 Same tasks, also from: Opus 5.5 series, wider run, three ways, 2026-09-26 Same tasks, also from: Fable 5, wider run, three ways, 2026-09-19
Other runs (3)
  • Second run, every task twice, 2026-08-31 Claude Opus 5 88.5% (tier 1), Claude Fable 5 88.0% (tier 1), Claude Sonnet 5 59.0% (tier 1), Claude Haiku 4.5 23.1% (tier 1)
  • First run, single attempt, 2026-08-31 Claude Fable 5 92.3% (tier 1), Claude Opus 5 87.9% (tier 1)
  • Fable 5.1, single attempt, 2026-09-03 Claude Fable 5.1 98.3% (tier 1)
23.1% tier 193.2% tier 1 too close to call same tasks, another run97.2% tier 1 best88.1% tier 1 too close to call93.3% tier 1 too close to call same tasks, another run53.0% tier 1
Skill authoring Wider run, three ways, 2026-09-06 Same tasks, also from: Opus 5.5 series, wider run, three ways, 2026-09-26 Same tasks, also from: Fable 5, wider run, three ways, 2026-09-19
Other runs (3)
  • Second run, every task twice, 2026-08-31 Claude Fable 5 100.0% (tier 1), Claude Opus 5 100.0% (tier 1), Claude Sonnet 5 100.0% (tier 1), Claude Haiku 4.5 70.0% (tier 1)
  • First run, single attempt, 2026-08-31 Claude Fable 5 100.0% (tier 1), Claude Opus 5 100.0% (tier 1)
  • Fable 5.1, single attempt, 2026-09-03 Claude Fable 5.1 100.0% (tier 1)
67.3% tier 1100.0% tier 1 best same tasks, another run100.0% tier 1 best100.0% tier 1 best99.8% tier 1 too close to call same tasks, another run100.0% tier 1 best
Spec writing Wider run, three ways, 2026-09-06 Same tasks, also from: Opus 5.5 series, wider run, three ways, 2026-09-26 Same tasks, also from: Fable 5, wider run, three ways, 2026-09-19
Other runs (3)
  • Second run, every task twice, 2026-08-31 Claude Haiku 4.5 98.6% (tier 1), Claude Fable 5 95.7% (tier 1), Claude Sonnet 5 94.8% (tier 1), Claude Opus 5 92.5% (tier 1)
  • First run, single attempt, 2026-08-31 Claude Fable 5 93.2% (tier 1), Claude Opus 5 92.3% (tier 1)
  • Fable 5.1, single attempt, 2026-09-03 Claude Fable 5.1 97.7% (tier 1)
99.2% tier 1 too close to call95.9% tier 1 same tasks, another run97.9% tier 1 too close to call92.3% tier 199.5% tier 1 best same tasks, another run96.2% tier 1
Link graph and metadata parity audit Workflow run, five times each, 2026-09-10 No range was published, so none is marked as close.not run *not run100.0% tier 1 best92.9% tier 1not run73.7% tier 1
Corpus integrity and correction Workflow run, five times each, 2026-09-10 No range was published, so none is marked as close.not run *not run71.7% tier 175.8% tier 1 bestnot run64.2% tier 1
Traffic drop triage Workflow run, five times each, 2026-09-10 No range was published, so none is marked as close.not run *not run100.0% tier 1 best100.0% tier 1 bestnot run80.0% tier 1
Getting clean JSON back Harder tasks, tier 2, 2026-09-19 Same tasks, also from: Claude Opus 5.5, tier 2, 2026-09-25 No model managed this on its own.
Other runs (3)
  • Main run, 2026-09-03 Claude Fable 5 0.0% (tier 1), Claude Fable 5.1 0.0% (tier 1), Claude Haiku 4.5 0.0% (tier 1), Claude Opus 5 0.0% (tier 1), Claude Sonnet 5 0.0% (tier 1)
  • Opus 5.5 series, each task asked once, 2026-09-25 Claude Opus 5.5 0.0% (tier 1)
  • Claude Opus 5.5, asked each task five times, 2026-09-28 Claude Opus 5.5 0.0% (tier 1)
0.0% tier 2in another run0.0% tier 20.0% tier 20.0% tier 2 same tasks, another run0.0% tier 2
Getting the length right Harder tasks, tier 2, 2026-09-19 Same tasks, also from: Claude Opus 5.5, tier 2, 2026-09-25
Other runs (3)
  • Main run, 2026-09-03 Claude Fable 5 76.0% (tier 1), Claude Fable 5.1 50.2% (tier 1), Claude Haiku 4.5 27.5% (tier 1), Claude Sonnet 5 17.8% (tier 1), Claude Opus 5 15.0% (tier 1)
  • Opus 5.5 series, each task asked once, 2026-09-25 Claude Opus 5.5 80.1% (tier 1)
  • Claude Opus 5.5, asked each task five times, 2026-09-28 Claude Opus 5.5 76.6% (tier 1)
4.2% tier 2 bestin another run0.3% tier 2 too close to call0.3% tier 2 too close to call0.3% tier 2 too close to call same tasks, another run1.4% tier 2 too close to call
Questions about recent events Main run, 2026-09-03
Other runs (2)
  • Opus 5.5 series, each task asked once, 2026-09-25 Claude Opus 5.5 100.0% (tier 1)
  • Claude Opus 5.5, asked each task five times, 2026-09-28 Claude Opus 5.5 93.3% (tier 1)
96.7% tier 1 best87.8% tier 1 too close to call92.2% tier 1 too close to call80.3% tier 1 too close to callin another run93.3% tier 1 too close to call
Better step-by-step answers Main run, 2026-09-03 Same tasks, also from: Claude Opus 5.5, asked each task five times, 2026-09-28
Other runs (1)
  • Opus 5.5 series, each task asked once, 2026-09-25 Claude Opus 5.5 40.0% (tier 1)
20.0% tier 1 too close to call40.0% tier 1 best40.0% tier 1 best40.0% tier 1 best40.0% tier 1 best same tasks, another run40.0% tier 1 best
  • * Claude Haiku 4.5 on Link graph and metadata parity audit, Corpus integrity and correction and Traffic drop triage. Before we buy a single run we check that a session can only read the files we put in front of it. That check never came back the way it has to, on any of the 3 tries we had written down beforehand, so we bought no runs at all. This is us declining to measure, and it says nothing about the model.

In Codex

JobGPT-5.4 miniGPT-5.6 LunaGPT-5.6 TerraGPT-6 LunaGPT-6 Sol
Accessibility audit Not run this waynot runnot runnot runnot runnot run
Code review Not run this waynot runnot runnot runnot runnot run
On-page audit Not run this waynot runnot runnot runnot runnot run
Skill authoring Not run this waynot runnot runnot runnot runnot run
Spec writing Not run this waynot runnot runnot runnot runnot run
Link graph and metadata parity audit Not run this waynot runnot runnot runnot runnot run
Corpus integrity and correction Not run this waynot runnot runnot runnot runnot run
Traffic drop triage Not run this waynot runnot runnot runnot runnot run
Getting clean JSON back Codex run, tip by tip, 2026-10-03 No model managed this on its own.0.0% tier 10.0% tier 10.0% tier 10.0% tier 10.0% tier 1
Getting the length right Codex run, tip by tip, 2026-10-03 No range was published, so none is marked as close.
Other runs (1)
  • Codex run, replaying two earlier tests, 2026-09-07 GPT-5.4 mini 18.6% (tier 1)
20.6% tier 157.7% tier 162.4% tier 1 best51.0% tier 1withheld *
Questions about recent events Codex run, tip by tip, 2026-10-03 One model on this run, so none is marked.
Other runs (1)
  • Codex run, replaying two earlier tests, 2026-09-07 GPT-5.4 mini 15.0% (tier 1)
withheld *withheld *30.0% tier 1withheld *withheld *
Better step-by-step answers Codex run, tip by tip, 2026-10-03 No range was published, so none is marked as close.30.0% tier 130.0% tier 130.0% tier 130.0% tier 140.0% tier 1 best
  • * GPT-6 Sol on Getting the length right. Incomplete coverage at retry exhaustion; 1 registered units have no valid record. Resume only within registered retry policy.
  • * GPT-5.4 mini on Questions about recent events. Model no longer served to this account after 38 of 60 units; two retries refused.
  • * GPT-5.6 Luna on Questions about recent events. 6 units exhausted the single C8 retry after policy-blocked built-in tool attempts; no completed tool or MCP hit. Full one-pass coverage not reached.
  • * GPT-6 Luna on Questions about recent events. Incomplete coverage at retry exhaustion; 9 registered units have no valid record. Resume only within registered retry policy.
  • * GPT-6 Sol on Questions about recent events. Incomplete coverage at retry exhaustion; 15 registered units have no valid record. Resume only within registered retry policy.

Every model, and what tested it

Every count here comes from the saved test results. A model is listed because a test produced a result for it, not because one was planned.

Models with at least one committed cell, in matrix order.
ModelInstrumentsVersionsTask sets measuredPage
Claude Haiku 4.5tested via API; tested in Claude Code120Claude Haiku 4.5
GPT-5 minitested via API116GPT-5 mini
Gemini 3.1 Flash Litetested via API116Gemini 3.1 Flash Lite
Claude Fable 5tested in Claude Code2 Claude Fable 5 and Claude Fable 5.117Claude Fable 5
Claude Fable 5.1tested in Claude Code2 Claude Fable 5 and Claude Fable 5.117Claude Fable 5.1
Claude Opus 5tested in Claude Code117Claude Opus 5
Claude Opus 5.5tested in Claude Code117Claude Opus 5.5
Claude Sonnet 5tested in Claude Code117Claude Sonnet 5
GPT-5.4 minitested via API; tested in Codex112GPT-5.4 mini
GPT-5.6 Lunatested in Codex111GPT-5.6 Luna
GPT-5.6 Terratested in Codex112GPT-5.6 Terra
GPT-6 Lunatested in Codex111GPT-6 Luna
GPT-6 Soltested in Codex110GPT-6 Sol

A model line with more than one served version renders as a series: each version keeps its own row on every table, and the per-task-set shifts between them are published on the comparison page and on each version’s own page. Versions are never collapsed into one row.