Better step-by-step answers

Does 'think step by step' help?

Ask the model to work through the problem step by step before it answers.

Why it matters

Multi-step problems are where models slip. Working through them first catches errors early.

How to use it

Add a line asking for the steps before the answer on any task with chained arithmetic or logic. Check that it helps on the model you use.

Holds

Pass rate up 20 points on Claude Haiku 4.5 and 34 points on GPT-5 mini. Inside the margin on Gemini 3.1 Flash Lite. Those results are tested via API. Also measured in Claude Code and Codex, reported separately on this page and never averaged with this.

tested via API · Holds on Claude Haiku 4.5, tested via API · Holds on GPT-5 mini, tested via API · No measured effect on Gemini 3.1 Flash Lite, tested via API

tested in Claude Code · No measured effect on Claude Fable 5, tested in Claude Code · No measured effect on Claude Opus 5, tested in Claude Code · Holds on Claude Haiku 4.5, tested in Claude Code · No measured effect on Claude Sonnet 5, tested in Claude Code · No measured effect on Claude Fable 5.1, tested in Claude Code

tested in Codex · No measured effect on GPT-5.4 mini, Codex run on GPT-5.4 mini, tested in Codex · No measured effect on GPT-5.6 Luna, Codex run on GPT-5.6 Luna, tested in Codex · No measured effect on GPT-5.6 Terra, Codex run on GPT-5.6 Terra, tested in Codex

What the marks mean

  • Holds
  • No measured effect
See the numbers per model

Without to with, per model

Via API tested via API

Claude Haiku 4.5

0.300
0.500

Holds

Gemini 3.1 Flash Lite

0.400
0.500

No measured effect

GPT-5 mini

0.160
0.500

Holds

deterministic pass rate, 0 to 1

In Claude Code tested in Claude Code

Claude Fable 5

0.400
0.500

No measured effect

Claude Fable 5.1

0.400
0.500

No measured effect

Claude Haiku 4.5

0.200
0.500

Holds

Claude Sonnet 5

0.400
0.500

No measured effect

Claude Opus 5

0.400
0.460

No measured effect

deterministic pass rate, 0 to 1

A picture of the per-model numbers, drawn from the same results. The tables are the source. Grey is the score without. Colour is the score with. The bracket shows how much the difference could move if we ran it again. Rows are ordered by the score with. A model measured at two versions keeps its versions next to each other. The two test methods are reported separately and never averaged.
Show per-task detail

Every item, without and with, per model

Via API tested via API

  • GPT-5 mini
  • Claude Haiku 4.5
  • Gemini 3.1 Flash Lite

deterministic pass rate, 0 to 1

Hover or focus an item to read its task and every model’s two values.

Pooled: the interval is taken over the records, not these means. These are those same records collapsed per item, so where an item carries more passes than another the lines need not average to the figure the table prints.

The item means behind this plot
Per-item arm means. Without is the unaided arm, with is the treated arm.
ItemGPT-5 mini Claude Haiku 4.5 Gemini 3.1 Flash Lite
Task without with without with without with
t01 0.000 0.000 0.000 0.000 0.000 0.000
t02 0.000 0.000 0.000 0.000 0.000 0.000
t04 0.000 0.000 0.000 0.000 0.000 0.000
t05 0.000 0.000 0.000 0.000 0.000 0.000
t07 0.000 1.000 0.000 1.000 0.000 1.000
t09 0.000 0.000 0.000 0.000 0.000 0.000
t03 0.200 1.000 0.000 1.000 1.000 1.000
t06 0.200 1.000 1.000 1.000 1.000 1.000
t10 0.200 1.000 1.000 1.000 1.000 1.000
t08 1.000 1.000 1.000 1.000 1.000 1.000

In Claude Code tested in Claude Code

  • Claude Haiku 4.5
  • Claude Fable 5
  • Claude Fable 5.1
  • Claude Opus 5
  • Claude Sonnet 5

deterministic pass rate, 0 to 1

Hover or focus an item to read its task and every model’s two values.

Pooled: the interval is taken over the records, not these means. These are those same records collapsed per item, so where an item carries more passes than another the lines need not average to the figure the table prints.

The item means behind this plot
Per-item arm means. Without is the unaided arm, with is the treated arm.
ItemClaude Haiku 4.5 Claude Fable 5 Claude Fable 5.1 Claude Opus 5 Claude Sonnet 5
Task without with without with without with without with without with
t01 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
t02 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
t04 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
t05 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
t07 0.000 1.000 0.000 1.000 0.000 1.000 0.000 0.600 0.000 1.000
t09 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
t03 0.200 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
t08 0.200 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
t06 0.600 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
t10 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
A picture of the per-item means behind the per-model numbers. The tables are the source. Each model draws two lines over the same task set: a dashed line through its unaided scores and a solid line through its treated ones. Items are ordered by the mean unaided score across the models that measured them, lowest first. The two instruments are reported separately and never averaged.

Result

The phrase think step by step raised the pass rate by 20 points on Claude Haiku 4.5 and 34 points on GPT-5 mini tested via API while staying inside the margin on Gemini 3.1 Flash Lite, and showed no measured effect on Claude Fable 5, Claude Opus 5, Claude Sonnet 5 and Claude Fable 5.1; held on Claude Haiku 4.5 tested in Claude Code, and showed no measured effect on GPT-5.4 mini via the Codex run on GPT-5.4 mini, and showed no measured effect on GPT-5.6 Luna via the Codex run on GPT-5.6 Luna, and showed no measured effect on GPT-5.6 Terra via the Codex run on GPT-5.6 Terra. Measured 2026-08-12.

Show the per-model numbers

Claim tested: The phrase think step by step improves accuracy on multi-step word problems.

Holds on Claude Haiku 4.5 and GPT-5 mini. No measured effect on Gemini 3.1 Flash Lite.

Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.

This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.

Ledger idC05-think-step-by-step

Clarification, . Until 2026-08-31 this figure read 5, which was the pass count rather than a number of calls. What changed, and why it is filed once.

What was tested

This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.

The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.

Per-model numbers

Per-model results. Means are over valid scored records only. Invalid records are excluded from every denominator and counted in coverage.
MeasureClaude Haiku 4.5GPT-5 miniGemini 3.1 Flash Lite
Control arm0.300n 500.160n 500.400n 50
Treatment arm0.500n 500.500n 500.500n 50
Delta+0.200+0.340+0.100
Interval, 95 percent0.010 to 0.3900.166 to 0.514-0.096 to 0.296
OrbitStable100 of 100 recordsStable100 of 100 recordsIn free drift100 of 100 records

Orbit is assigned by the frozen status_v1 rule. On this scale, deterministic pass rate, 0 to 1, the pass threshold is +0.20 and the failure floor is -0.20, each requiring an interval that excludes zero.

In Claude Code

These cells are tested in Claude Code, on a subscription path with no API key. They are a second instrument and are never averaged with the figures above, which are tested via API. What that means, and how it was calibrated.

Single-turn cells, replayed from the committed claims on the Claude Code CLI.
ModelControlTreatmentDeltaIntervalPairsOrbit
Claude Fable 50.40000.5000+0.100-0.096 to 0.29650In free drift
Claude Opus 50.40000.4600+0.060-0.136 to 0.25650In free drift
Claude Haiku 4.50.20000.5000+0.3000.121 to 0.47950Stable
Claude Sonnet 50.40000.5000+0.100-0.096 to 0.29650In free drift
Claude Fable 5.10.40000.5000+0.100-0.096 to 0.29650In free drift

Method for this claim

Task set
10 arithmetic word problems requiring two or three chained operations. Every answer was computed by hand and is a whole number. Problem text is identical between arms.
Runs per model per arm
5 passes x 10 items
Scoring
Deterministic, via scoreExactMatch. A committed function scores each answer with no model in the loop.
Pass criterion as written for the pilot
Treatment exact-match rate exceeds control by at least 10 percentage points on the same 10 problems.

The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.

Model versions, as recorded

Read from the run records, not from configuration.
ModelVersion string returnedDelta on this claimInterval
Claude Haiku 4.5claude-haiku-4-5-20251001+0.2000.010 to 0.390
GPT-5 minigpt-5-mini-2025-08-07+0.3400.166 to 0.514
Gemini 3.1 Flash Litegemini-3.1-flash-lite+0.100-0.096 to 0.296

Reading across models

Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.

GPT-5 mini rejected the fixed sampling setting and ran at its own default on all 1,040 of its calls. The other models ran at temperature 0.

The tip above is editorial. Every figure inside the measurement is computed at build time from the committed pilot records by the status_v1 rule, and none of it is written by hand.