Model under test
GPT-5.4 mini
Every result this site holds for GPT-5.4 mini. We tested this model two ways, through the API and inside Codex, and never average them.
Version strings recorded, as returned: gpt-5.4-mini, gpt-5.4-mini-2026-03-17. One answer names a dated build: gpt-5.4-mini-2026-03-17. That is the vendor resolving the name we asked for to its own build of it, so it is the model we asked for, written down. For the results tested in Codex, we cannot say which model answered. That test method returns no name for whatever produced the answer, so there is nothing to check against the name we asked for. The same work was also run against this model through the API, which does name what answered, so the two readings can be set beside each other.
What was measured, and where
We ran 15 separate tests on this model, listed by test method below and never added together. A test run through the API and inside Codex are two different measurements, so a single figure across them would be one no run produced. How the test methods were compared.
Prompting tips, replayed through the API beside Codex
2 tests, tested via API, API twin for the Codex run.
- Helped 1 of 2
- Holds 1 of 2
Prompting tips, replayed in Codex
2 tests, tested in Codex, Codex run, replaying two earlier tests.
- Helped 1 of 2
- Holds 1 of 2
Prompting tips, replayed in Codex
11 tests, tested in Codex, Codex run on GPT-5.4 mini.
- Helped 5 of 11
- Could not measure 5 of 11
- No measured effect 1 of 11
One tip was measured and not published. There is no result for it, and none is estimated from the answers that did arrive.
- Questions about recent events: model no longer served to this account after 38 of 60 units; two retries refused
What these figures can be set beside
Every table on this page is one job, run one way, on one occasion. A model tested the other way is not on those tables and its figures are not set against these. We checked the two test methods against each other and they came out different, so they are reported side by side and never averaged.
- Measured on
- tested via API; tested in Codex.
- Not directly comparable
- Claude Fable 5; Claude Fable 5.1; Claude Opus 5; Claude Opus 5.5; Claude Sonnet 5 were tested inside Claude Code, which this model did not run. No figure below is theirs and none of theirs is set against these.
This model is what connects through the API and inside Codex
GPT-5.4 mini was tested through the API and inside Codex, so its own readings are what connect them. They are listed job by job below and are never combined into one figure. It is one of two models that connect a pair of test methods this way. Each connects its own pair, and no chain is drawn through them.
Show the readings, job by job (12 jobs)
- Getting clean JSON back GPT-5.4 mini scored 0.0% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Instructions before or after GPT-5.4 mini scored 100.0% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Using tags and formatting GPT-5.4 mini scored 100.0% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Showing examples GPT-5.4 mini scored 100.0% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Showing examples GPT-5.4 mini scored 80.0% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Showing examples GPT-5.4 mini scored 78.6% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Better step-by-step answers GPT-5.4 mini scored 30.0% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Stop made-up answers GPT-5.4 mini scored 30.0% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Instructions in long prompts GPT-5.4 mini scored 100.0% (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Offering the model money GPT-5.4 mini scored 193 words (no bracket published) tested in Codex, Codex run, tip by tip. That is one test method only, so it bridges nothing on its own.
- Questions about recent events GPT-5.4 mini scored 27.5% (no bracket published) tested via API, API twin for the Codex run, and 15.0% (no bracket published) tested in Codex, Codex run, replaying two earlier tests. They are stated side by side and are not differenced. How well the two paths agree here: continuous with the API twin, 2 of 2 tests overlap. What was compared.
- Getting the length right GPT-5.4 mini scored 15.7% (no bracket published) tested via API, API twin for the Codex run, and 18.6% (no bracket published) tested in Codex, Codex run, replaying two earlier tests, and 20.6% (no bracket published) tested in Codex, Codex run, tip by tip. They are stated side by side and are not differenced. How well the two paths agree here: continuous with the API twin, 2 of 2 tests overlap. What was compared.
- The grader, separately
- a second test method, 92 of 120 picks, 10 of 12 tests. The grader itself, replayed on the committed judgements rather than on any subject. Both criteria we set in advance failed, and the graded classes stayed unrun on this path because of it.
What this model does on its own, one job at a time
We measured this model on 12 kinds of work, and the scores for every one of them are below.
Show the figures, one job at a time
One table per job and per test method, ranked by what the model did with no help. Nothing here is averaged across a job, a test method or a run. Every model’s tables sit together on the comparison page.
GPT-5.4 mini placed 4 of 4 on this work.
What one task costs with no help, and what it scores Getting the length right
- tested via API
- tested in Claude Code
Across: what one task costs with no help, in US dollars at list price. Each gridline is ten times the one before it. Up: deterministic pass rate, 0 to 1.
- Gemini 3.1 Flash Lite Main run, tip by tip tested via API $0.00513 per task scored 31.0%, 5.8% to 56.1%
- GPT-5 mini Main run, tip by tip tested via API $0.01006 per task scored 1.6%, 0.0% to 4.6%†
- Claude Haiku 4.5 Main run, tip by tip tested via API $0.01135 per task scored 24.7%, 1.2% to 48.2%
- Claude Opus 5.5 Opus 5.5 series, each task asked once tested in Claude Code $0.01233 per task scored 80.1%, 62.5% to 97.8% 16.3 s per call, middle half 15.0 s to 19.9 s
- Claude Haiku 4.5 tested in Claude Code $0.01432 per task scored 27.5%, 0.0% to 54.9%
- GPT-5.4 mini API twin for the Codex run tested via API $0.01442 per task scored 15.7%, no range published
- Claude Sonnet 5 tested in Claude Code $0.06459 per task scored 17.8%, 0.0% to 41.0%†
- Claude Fable 5 tested in Claude Code $0.16668 per task scored 76.0%, 53.7% to 98.4%
- Claude Opus 5 tested in Claude Code $0.18907 per task scored 15.0%, 0.0% to 35.1%†
- Claude Fable 5.1 tested in Claude Code $0.22184 per task scored 50.2%, 21.9% to 78.5%
- † The bracket runs past the end of the scale. It is drawn to the end; the width beyond it is real and is what a small number of items buys.
- Marks here are all one size. Not every mark has a time behind it. Sizing only the ones that do would draw the rest at a size we did not measure. Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Measured, and not on this chart
- GPT-5.4 mini Codex run, replaying two earlier tests tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
- GPT-5.4 mini Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
- GPT-5.6 Luna Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
- GPT-5.6 Terra Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
- GPT-6 Luna Codex run, tip by tip tested in Codex This test did not report what a task cost on this model, so there is no place for it on the cost scale. It is left off rather than drawn at a guess. Why some figures are blank.
This is one of 12 sets of work this model was measured on. The same chart for every one of them sits on the comparison page, and the tables below carry this model’s figures for all of them.
Getting clean JSON back
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 0.0% interval not measured 10 items | 100.0% C01-json-schema | not measured | 3,879 in, 49 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Luna | 0.0% interval not measured 10 items | 100.0% C01-json-schema | not measured | 4,620 in, 53 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Terra | 0.0% interval not measured 10 items | 100.0% C01-json-schema | not measured | 6,135 in, 44 out | 0 | one pass, not measured | not measured |
| 1 | GPT-6 Luna | 0.0% interval not measured 10 items | 100.0% C01-json-schema | not measured | 6,679 in, 51 out | 0 | one pass, not measured | 9.5 s 8.2 s to 10.7 s across the middle half of 20 calls |
| 1 | GPT-6 Sol | 0.0% interval not measured 10 items | 100.0% C01-json-schema | not measured | 6,659 in, 44 out | 0 | one pass, not measured | 8.4 s 8.0 s to 10.5 s across the middle half of 20 calls |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Instructions before or after
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 100.0% interval not measured 10 items | 100.0% C02-instruction-after | not measured | 4,010 in, 9 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Luna | 100.0% interval not measured 10 items | 100.0% C02-instruction-after | not measured | 4,754 in, 9 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Terra | 100.0% interval not measured 10 items | 100.0% C02-instruction-after | not measured | 6,248 in, 9 out | 0 | one pass, not measured | not measured |
| 1 | GPT-6 Luna | 100.0% interval not measured 10 items | 100.0% C02-instruction-after | not measured | 6,882 in, 9 out | 0 | one pass, not measured | 7.2 s 6.8 s to 8.0 s across the middle half of 20 calls |
| 1 | GPT-6 Sol | 100.0% interval not measured 10 items | 100.0% C02-instruction-after | not measured | 6,812 in, 9 out | 0 | one pass, not measured | 7.8 s 7.0 s to 8.9 s across the middle half of 20 calls |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Using tags and formatting
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 100.0% interval not measured 10 items | 100.0% C03-xml-delimiters | not measured | 3,973 in, 37 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Luna | 100.0% interval not measured 10 items | 90.0% C03-xml-delimiters | not measured | 4,715 in, 35 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Terra | 100.0% interval not measured 10 items | 100.0% C03-xml-delimiters | not measured | 6,230 in, 33 out | 0 | one pass, not measured | not measured |
| 1 | GPT-6 Luna | 100.0% interval not measured 10 items | 90.0% C03-xml-delimiters | not measured | 6,773 in, 35 out | 0 | one pass, not measured | 8.0 s 7.5 s to 8.7 s across the middle half of 20 calls |
| 1 | GPT-6 Sol | 100.0% interval not measured 10 items | 100.0% C03-xml-delimiters | not measured | 6,774 in, 43 out | 0 | one pass, not measured | 8.9 s 8.3 s to 9.5 s across the middle half of 20 calls |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Showing examples
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 100.0% interval not measured 10 items | 100.0% C04-three-shot-format | not measured | 3,894 in, 22 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Luna | 100.0% interval not measured 10 items | 100.0% C04-three-shot-format | not measured | 4,637 in, 22 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Terra | 100.0% interval not measured 10 items | 100.0% C04-three-shot-format | not measured | 6,173 in, 22 out | 0 | one pass, not measured | not measured |
| 1 | GPT-6 Luna | 100.0% interval not measured 10 items | 100.0% C04-three-shot-format | not measured | 6,719 in, 22 out | 0 | one pass, not measured | 7.3 s 7.1 s to 8.0 s across the middle half of 20 calls |
| 1 | GPT-6 Sol | 100.0% interval not measured 10 items | 100.0% C04-three-shot-format | not measured | 6,718 in, 22 out | 0 | one pass, not measured | 8.0 s 7.6 s to 8.2 s across the middle half of 20 calls |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Showing examples
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.6 Luna | 82.9% interval not measured 10 items | 95.7% C04b-zero-vs-one | not measured | 4,720 in, 32 out | 0 | one pass, not measured | not measured |
| 2 | GPT-5.6 Terra | 81.4% interval not measured 10 items | 100.0% C04b-zero-vs-one | not measured | 6,214 in, 31 out | 0 | one pass, not measured | not measured |
| 2 | GPT-6 Luna | 81.4% interval not measured 10 items | 95.7% C04b-zero-vs-one | not measured | 6,778 in, 31 out | 0 | one pass, not measured | 7.6 s 7.3 s to 8.2 s across the middle half of 20 calls |
| 2 | GPT-6 Sol | 81.4% interval not measured 10 items | 98.6% C04b-zero-vs-one | not measured | 6,847 in, 31 out | 0 | one pass, not measured | 8.0 s 7.7 s to 8.7 s across the middle half of 20 calls |
| 3 | GPT-5.4 mini | 80.0% interval not measured 10 items | 100.0% C04b-zero-vs-one | not measured | 3,956 in, 31 out | 0 | one pass, not measured | not measured |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Showing examples
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.6 Luna | 81.4% interval not measured 10 items | 100.0% C04b-zero-vs-three | not measured | 4,719 in, 33 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Terra | 81.4% interval not measured 10 items | 100.0% C04b-zero-vs-three | not measured | 6,215 in, 31 out | 0 | one pass, not measured | not measured |
| 1 | GPT-6 Luna | 81.4% interval not measured 10 items | 100.0% C04b-zero-vs-three | not measured | 6,778 in, 32 out | 0 | one pass, not measured | 7.3 s 7.2 s to 7.5 s across the middle half of 20 calls |
| 1 | GPT-6 Sol | 81.4% interval not measured 10 items | 100.0% C04b-zero-vs-three | not measured | 6,779 in, 31 out | 0 | one pass, not measured | 8.0 s 7.7 s to 8.5 s across the middle half of 20 calls |
| 2 | GPT-5.4 mini | 78.6% interval not measured 10 items | 100.0% C04b-zero-vs-three | not measured | 3,956 in, 31 out | 0 | one pass, not measured | not measured |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Better step-by-step answers
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Sol | 40.0% interval not measured 10 items | 40.0% C05-think-step-by-step | not measured | 6,680 in, 5 out | 0 | one pass, not measured | 8.2 s 7.5 s to 8.9 s across the middle half of 20 calls |
| 2 | GPT-5.4 mini | 30.0% interval not measured 10 items | 40.0% C05-think-step-by-step | not measured | 3,879 in, 5 out | 0 | one pass, not measured | not measured |
| 2 | GPT-5.6 Luna | 30.0% interval not measured 10 items | 40.0% C05-think-step-by-step | not measured | 4,599 in, 5 out | 0 | one pass, not measured | not measured |
| 2 | GPT-5.6 Terra | 30.0% interval not measured 10 items | 40.0% C05-think-step-by-step | not measured | 6,135 in, 5 out | 0 | one pass, not measured | not measured |
| 2 | GPT-6 Luna | 30.0% interval not measured 10 items | 40.0% C05-think-step-by-step | not measured | 6,680 in, 5 out | 0 | one pass, not measured | 8.1 s 7.8 s to 8.8 s across the middle half of 20 calls |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Stop made-up answers
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 30.0% interval not measured 10 items | 90.0% C06-permit-idk | not measured | 3,857 in, 25 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Luna | 30.0% interval not measured 10 items | 100.0% C06-permit-idk | not measured | 4,599 in, 17 out | 0 | one pass, not measured | not measured |
| 1 | GPT-6 Sol | 30.0% interval not measured 10 items | 100.0% C06-permit-idk | not measured | 6,678 in, 19 out | 0 | one pass, not measured | 8.2 s 7.7 s to 8.5 s across the middle half of 20 calls |
| 2 | GPT-5.6 Terra | 20.0% interval not measured 10 items | 100.0% C06-permit-idk | not measured | 6,134 in, 15 out | 0 | one pass, not measured | not measured |
| 3 | GPT-6 Luna | 0.0% interval not measured 10 items | 100.0% C06-permit-idk | not measured | 6,679 in, 21 out | 0 | one pass, not measured | 7.7 s 7.1 s to 8.5 s across the middle half of 20 calls |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Instructions in long prompts
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 100.0% interval not measured 10 items | 100.0% C07-instruction-at-end | not measured | 4,086 in, 162 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Luna | 100.0% interval not measured 10 items | 100.0% C07-instruction-at-end | not measured | 4,829 in, 94 out | 0 | one pass, not measured | not measured |
| 1 | GPT-5.6 Terra | 100.0% interval not measured 10 items | 100.0% C07-instruction-at-end | not measured | 6,343 in, 100 out | 0 | one pass, not measured | not measured |
| 1 | GPT-6 Luna | 100.0% interval not measured 10 items | 100.0% C07-instruction-at-end | not measured | 6,887 in, 109 out | 0 | one pass, not measured | 9.8 s 9.0 s to 10.8 s across the middle half of 20 calls |
| 1 | GPT-6 Sol | 100.0% interval not measured 10 items | 100.0% C07-instruction-at-end | not measured | 6,887 in, 103 out | 0 | one pass, not measured | 10.2 s 9.7 s to 10.7 s across the middle half of 20 calls |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Offering the model money
tested in Codex, Codex run, tip by tip
| Rank | Model | Answer length, on its own | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 193 words interval not measured 10 items | 296 words C08-tip-length | not measured | 3,850 in, 260 out | 0 | one pass, not measured | not measured |
| 2 | GPT-5.6 Luna | 109 words interval not measured 10 items | 192 words C08-tip-length | not measured | 4,592 in, 148 out | 0 | one pass, not measured | not measured |
| 3 | GPT-5.6 Terra | 103 words interval not measured 10 items | 185 words C08-tip-length | not measured | 6,106 in, 138 out | 0 | one pass, not measured | not measured |
| 4 | GPT-6 Luna | 89 words interval not measured 10 items | 127 words C08-tip-length | not measured | 6,718 in, 116 out | 0 | one pass, not measured | 11.4 s 10.5 s to 14.0 s across the middle half of 20 calls |
| 5 | GPT-6 Sol | 79 words interval not measured 10 items | 127 words C08-tip-length | not measured | 6,650 in, 105 out | 0 | one pass, not measured | 10.9 s 10.2 s to 12.2 s across the middle half of 20 calls |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Questions about recent events
tested via API, API twin for the Codex run
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 27.5% interval not measured 30 items | 86.7% C16-cutoff-disclosure | $0.00048 billed through the API | 52 in, 97 out | not measured | 25 of 30 63.3 percentage points apart where not 56 more answered once at temperature 0 | not measured |
0 of the 108 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Questions about recent events
tested in Codex, Codex run, replaying two earlier tests
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 15.0% interval not measured 30 items | 69.2% C16-cutoff-disclosure | not measured | 19,960 in, 107 out | 0 | 19 of 30 62.1 percentage points apart where not 56 more answered once setting not recorded | not measured |
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Getting the length right
tested via API, API twin for the Codex run
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 15.7% interval not measured 10 items | 96.9% C17-exact-length | $0.01442 billed through the API | 139 in, 3,182 out | not measured | 8 of 20 3.5 percentage points apart where not at temperature 0 | not measured |
0 of the 70 answers behind this figure came back with a thinking-token count. A run that reported no count did not report zero.
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Getting the length right
tested in Codex, Codex run, replaying two earlier tests
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 mini | 18.6% interval not measured 10 items | 96.7% C17-exact-length | not measured | 45,193 in, 2,211 out | 0 | 6 of 20 6.4 percentage points apart where not setting not recorded | not measured |
Each answer here carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
Getting the length right
tested in Codex, Codex run, tip by tip
| Rank | Model | Unaided score | Best assisted score | Cost per task | Tokens per task | Thinking per task | Same answer on a repeat | Time per call |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.6 Terra | 62.4% interval not measured 10 items | 91.4% C17-exact-length | not measured | 6,108 in, 160 out | 0 | one pass, not measured | not measured |
| 2 | GPT-5.6 Luna | 57.7% interval not measured 10 items | 94.6% C17-exact-length | not measured | 4,593 in, 154 out | 0 | one pass, not measured | not measured |
| 3 | GPT-6 Luna | 51.0% interval not measured 10 items | 91.9% C17-exact-length | not measured | 6,652 in, 172 out | 0 | one pass, not measured | 11.2 s 9.5 s to 14.4 s across the middle half of 20 calls |
| 4 | GPT-5.4 mini | 20.6% interval not measured 10 items | 95.4% C17-exact-length | not measured | 3,851 in, 326 out | 0 | one pass, not measured | not measured |
Where the time is blank, each answer carries one time, the moment it was written down. There is no start and finish to measure a call between.
Nothing was billed for this run, and no price is estimated for it either. This test method does not report which model answered, so there is no published rate that is known to apply to it.
Assisted scores name the entry that produced them; skills and tips are ranked on their own pages, not here.
This model was also run through Codex, on a path that bills nothing. How it was called, what it was carrying, and what it cannot tell us, is below.
Show the full breakdown, tested in Codex
How this model was called, tested in Codex
Through the Codex command line, on a subscription with no API key and nothing billed. No figure on this page prices a Codex result, not even as an estimate: this path returns no name for the model that answered, so there is no published rate we know applies to it. What that means, and how it was checked.
- What the model was carrying
- tested in Codex (replaced base instructions; runtime tools, permissions and system skills present) Its base instructions were replaced with ours, and the rest of the Codex setup was still there. A result from this method and a result tested via API differ by that as well as by the tip being tested.
- Which model answered
- served model not reported by this instrument We asked for GPT-5.4 mini and the request was accepted. We looked for a name in the reply twice, two different ways, and there is none, so we cannot check it. On the method that does return one, we found 71 answers that came from a different model than the one we asked for.
These results are tested in Codex and are never averaged with any figure tested via API or tested in Claude Code. Where this model was measured on more than one, each is reported as its own panel above.