Does SEO onpage help? Tested on SEO tasks

SEO onpage, from rampstackco/claude-skills. Its on-page audit tasks were run with the skill loaded and without it. Each time it is the same task, run twice.

What we tested

Whether loading this skill helps a model do on-page SEO audits.

What counts as helping

The skill has to find at least 10 percentage points more of the planted problems than the model finds without it, on the same 10 test pages.

How we scored it

We planted 43 known problems across the 10 test pages and counted how many the model named. The score is the share it found, from 0 to 1.

Without to with, per model

Via API tested via API

Gemini 3.1 Flash Lite First run, with and without the skill

0.488

0.707

Holds

Gemini 3.1 Flash Lite Wider run, three ways

0.556

0.682

No measured effect

GPT-5 mini Wider run, three ways

0.495

0.575

No measured effect

GPT-5 mini First run, with and without the skill

0.450

0.492

No measured effect

Claude Haiku 4.5 First run, with and without the skill

0.239

0.474

No measured effect

deterministic pass rate, 0 to 1

In Claude Code tested in Claude Code

Claude Fable 5.1 Wider run, three ways

0.930

0.983

No measured effect

Claude Fable 5.1 Fable 5.1, single attempt

0.983

0.950

No measured effect

Claude Fable 5 Second run, every task twice

0.894

0.938

No measured effect

Claude Fable 5 First run, single attempt

0.905

0.925

No measured effect

Claude Opus 5 First run, single attempt

0.892

0.833

No measured effect

Claude Opus 5 Second run, every task twice

0.879

0.833

No measured effect

Claude Opus 5 Wider run, three ways

0.833

0.792

No measured effect

Claude Sonnet 5 Second run, every task twice

0.586

0.637

No measured effect

Claude Sonnet 5 Wider run, three ways

0.554

0.629

No measured effect

Claude Haiku 4.5 Wider run, three ways

0.435

0.411

No measured effect

Claude Haiku 4.5 Second run, every task twice

0.248

0.408

No measured effect

deterministic pass rate, 0 to 1

A picture of the per-model numbers, drawn from the same results. The tables are the source. Grey is the score without. Colour is the score with. The bracket shows how much the difference could move if we ran it again. Rows are ordered by the score with. A model measured at two versions keeps its versions next to each other. The two test methods are reported separately and never averaged. A bracket wider than the axis is drawn to the edge with its cap omitted; the table gives its bounds.

Show per-task detail

Every item, without and with, per model

Via API tested via API

Claude Haiku 4.5
GPT-5 mini
Gemini 3.1 Flash Lite
GPT-5 mini
Gemini 3.1 Flash Lite

deterministic pass rate, 0 to 1

Hover or focus an item to read its task and every model’s two values.

Task-paired: these item means are what this cell's interval is built from.

The item means behind this plot

Per-item arm means. Without is the unaided arm, with is the treated arm.
Item	Claude Haiku 4.5		GPT-5 mini		Gemini 3.1 Flash Lite		GPT-5 mini		Gemini 3.1 Flash Lite
Task	without	with	without	with	without	with	without	with	without	with
p08	0.000	0.000	0.000	0.500	0.000	0.000	0.000	0.000	0.000	0.000
p05	0.000	0.000	0.333	0.333	0.333	0.667	0.333	0.667	0.333	0.667
p07	0.000	0.750	0.000	0.000	0.500	0.750	0.250	1.000	0.750	0.500
p02	0.000	1.000	0.600	0.800	0.000	0.800	0.600	0.800	0.600	0.800
p04	0.333	0.667	0.333	0.333	0.500	0.833	0.667	0.333	0.333	0.833
p10	0.167	0.500	0.667	0.667	0.333	0.667	0.667	0.667	0.333	0.667
p06	0.000	0.000	0.500	0.750	0.500	0.750	0.750	0.750	0.500	0.500
p01	0.143	0.571	0.571	0.286	0.714	0.857	0.429	0.286	0.714	0.857
p03	0.750	0.750	0.500	0.750	1.000	0.750	0.750	0.750	1.000	1.000
p09	1.000	0.500	1.000	0.500	1.000	1.000	0.500	0.500	1.000	1.000

A picture of the per-item means behind the per-model numbers. The tables are the source. Each model draws two lines over the same task set: a dashed line through its unaided scores and a solid line through its treated ones. Items are ordered by the mean unaided score across the models that measured them, lowest first. The two instruments are reported separately and never averaged.

Without to with, per model

In Claude Code tested in Claude Code

Claude Haiku 4.5 Panel v2 (repeat sampling)

No measured effect

Claude Haiku 4.5 Expansion cohort (three arms)

No measured effect

Claude Sonnet 5 Expansion cohort (three arms)

No measured effect

Claude Sonnet 5 Panel v2 (repeat sampling)

No measured effect

Claude Opus 5 Expansion cohort (three arms)

No measured effect

Claude Opus 5 Panel v2 (repeat sampling)

No measured effect

Claude Opus 5 Skill cohort (one pass)

No measured effect

Claude Fable 5 Panel v2 (repeat sampling)

No measured effect

Claude Fable 5 Skill cohort (one pass)

No measured effect

Claude Fable 5.1 Expansion cohort (three arms)

No measured effect

Claude Fable 5.1 Fable 5.1, one pass

No measured effect

deterministic pass rate, 0 to 1

A picture of the per-model numbers, drawn from the same results. The tables are the source. Each row runs from the score without to the score with. The thin bar beneath it is the bracket, and it shows how much the difference could move if we ran it again. The two test methods are reported separately and never averaged. A bracket wider than the axis is drawn to the edge with its end cap omitted; the table gives its bounds.

The Claude Code runs committed their cells as arm means and intervals rather than as per-item values, so this panel is drawn without to with rather than item by item.

What exactly was tested, and how it was scored

Repository: rampstackco/claude-skills
Path: skills/seo-onpage
Commit: 0479242522549dfdb389bb9b7807ad4d6016ffb7
Content hash: 045b5110b9c9b993d172fb98dd6696b6cc87cb407d67226724957184ed6371c1
Date tested: 2026-08-30

The pin is the whole of this skill's identity here. It resolves at https://github.com/rampstackco/claude-skills/tree/0479242522549dfdb389bb9b7807ad4d6016ffb7/skills/seo-onpage, and the content hash is a sha256 over exactly the text the model was given with the skill loaded. Nothing else about the skill appears on this site.

`S07-seo-onpage`

Loading skills/seo-onpage from rampstackco/claude-skills improves outputs on on-page audit tasks.

Pass criterion: With-arm mean coverage exceeds the without-arm by at least 0.10 on the same 10 fixture pages at the same model and settings.
Scale: unit. Issues found over issues seeded, so a [0,1] fraction, which is the unit scale.
Task pairs planned per model: 10
Notes: 43 issues seeded across ten pages, ledger at fixtures/on-page/issue-ledger.json. The code set is closed and is given to both arms in the prompt, so the checklist is not the manipulated variable.

Injected context tokens

Injected context tokens, per arm
Arm	Context characters	Injected context tokens
without	0	0
with	21,800	5,117

The character count is exact: it is the length of the text the with arm is given, and the content hash above is a sha256 over that same text. The token figure is an estimate at 4.26 characters per token, the ratio the phase 1 run measured over 3,120 calls, and it is labelled an estimate until a run reports its own token counts. The without arm is given the identical prompt and nothing else, so its zero is a measurement rather than a missing value.

Verdict per model

Effect per model
Model	Reading	Orbit	Without -> with	Effect	95 percent interval	Task pairs	Model version returned
Claude Haiku 4.5	Deterministic, scored against a committed answer key	In free drift unclear. The model scored 24% without it.	0.2393 -> 0.4738	+0.2345	[-0.0323, 0.5014]	10 pairs, 20 of 20 records	`claude-haiku-4-5-20251001`
Gemini 3.1 Flash Lite	Deterministic, scored against a committed answer key	Stable	0.4881 -> 0.7074	+0.2193	[0.0464, 0.3922]	10 pairs, 20 of 20 records	`gemini-3.1-flash-lite`
GPT-5 mini	Deterministic, scored against a committed answer key	In free drift unclear. The model scored 45% without it.	0.4505 -> 0.4919	+0.0414	[-0.1350, 0.2179]	10 pairs, 20 of 20 records	`gpt-5-mini-2025-08-07`
Gemini 3.1 Flash Lite	Deterministic, scored against a committed answer key	In free drift (10 of 10 pairs) unclear. The model scored 49% without it.	0.5564 -> 0.6824	+0.1260	[-0.0105, 0.2624]	10 pairs, 30 of 38 records	`gemini-3.1-flash-lite`
GPT-5 mini	Deterministic, scored against a committed answer key	In free drift unclear. The model scored 58% without it.	0.4945 -> 0.5752	+0.0807	[-0.1017, 0.2632]	10 pairs, 30 of 30 records	`gpt-5-mini-2025-08-07`

In free drift: no separation the design can resolve. Not evidence of no effect.
Stable: the treatment arm outscored the control arm by more than the threshold, and the interval excludes zero.

Cost

Cost per task
Model	Arm	Tasks attempted	Mean input tokens	Mean output tokens	Cost per task	Cost ratio
Claude Haiku 4.5 served `claude-haiku-4-5-20251001`	without	10	667	37	$0.00085
Claude Haiku 4.5 served `claude-haiku-4-5-20251001`	with	10	6,602	58	$0.00689	8.08x
GPT-5 mini served `gpt-5-mini-2025-08-07`	without	10	566	74	$0.00029
GPT-5 mini served `gpt-5-mini-2025-08-07`	with	10	5,786	74	$0.00159	5.51x
Gemini 3.1 Flash Lite	without	10	599	53	$0.00023
Gemini 3.1 Flash Lite	with	10	6,165	50	$0.00162	7.06x

Defined in the metrics canon. Cost per task divides every dollar spent on an arm by the tasks attempted on it, including tasks whose call returned nothing, because a call that returned nothing was still billed.

Where the with arm did not win

Claude Haiku 4.5, Deterministic, scored against a committed answer key

Lost on 1 of 10 pairs: p09 (-0.5000).

Drew on 4 of 10 pairs: p03, p05, p06, p08.

Findings not in the answer key: 27 with the skill, 17 without. These are counted and never netted off coverage.

Gemini 3.1 Flash Lite, Deterministic, scored against a committed answer key

Lost on 1 of 10 pairs: p03 (-0.2500).

Drew on 2 of 10 pairs: p08, p09.

Findings not in the answer key: 15 with the skill, 26 without. These are counted and never netted off coverage.

GPT-5 mini, Deterministic, scored against a committed answer key

Lost on 2 of 10 pairs: p09 (-0.5000), p01 (-0.2857).

Drew on 4 of 10 pairs: p04, p05, p07, p10.

Findings not in the answer key: 53 with the skill, 54 without. These are counted and never netted off coverage.

Gemini 3.1 Flash Lite, Deterministic, scored against a committed answer key

Lost on 1 of 10 pairs: p07 (-0.2500).

Drew on 4 of 10 pairs: p03, p06, p08, p09.

Findings not in the answer key: 16 with the skill, 26 without. These are counted and never netted off coverage.

GPT-5 mini, Deterministic, scored against a committed answer key

Lost on 2 of 10 pairs: p04 (-0.3333), p01 (-0.1429).

Drew on 5 of 10 pairs: p03, p06, p08, p09, p10.

Findings not in the answer key: 52 with the skill, 56 without. These are counted and never netted off coverage.

In Claude Code

These cells are tested in Claude Code, on a subscription path with no API key. They are a second instrument: no figure here is averaged with one tested via API above. How the two were compared.

Claude Fable 5 · unclear, and both runs agree
Claude Opus 5 · unclear, and both runs agree
Claude Haiku 4.5 · unclear under repeat sampling
Claude Sonnet 5 · unclear under repeat sampling
Claude Fable 5.1 · unclear in the version re-test

Skill cohort (one pass)

one pass over each committed item. Matrix hash 2707a06a. No pre-registration: this run predates the practice on this arm. Run report.

One row per model. Every figure is read from this run’s committed cells.
Model	No skill	With skill	Delta	Interval	Pairs	Orbit	Injected tokens	False positives per page
Claude Fable 5	0.9048	0.9250	+0.020	-0.064 to 0.105	10	In free drift	8,220	7 with, 9 without
Claude Opus 5	0.8917	0.8333	-0.058	-0.187 to 0.071	10	In free drift	8,220	11 with, 7 without

Thinking tokens, recorded per arm. Reported, never scored: no interval or Orbit on this page consults them.
Model	Arm	Records that reasoned	Mean thinking tokens	Most on one record
Claude Fable 5	with skill	20 of 20	609	1,138
Claude Fable 5	no skill	20 of 20	483	931

Panel v2 (repeat sampling)

two passes over each committed item, under repeat sampling. Matrix hash 94bab960. Pre-registration · Run report.

One row per model. Every figure is read from this run’s committed cells.
Model	No skill	With skill	Delta	Interval	Pairs	Orbit	Injected tokens
Claude Fable 5	0.8940	0.9375	+0.043	-0.000 to 0.087	10	In free drift	not measured: 40 of 40 records predate the usage-capture amendment
Claude Haiku 4.5	0.2476	0.4085	+0.161	0.035 to 0.286	10	In free drift	not measured: 40 of 40 records predate the usage-capture amendment
Claude Opus 5	0.8792	0.8333	-0.046	-0.113 to 0.021	10	In free drift	not measured: 40 of 40 records predate the usage-capture amendment
Claude Sonnet 5	0.5860	0.6371	+0.051	-0.039 to 0.141	10	In free drift	not measured: 40 of 40 records predate the usage-capture amendment

Thinking tokens, recorded per arm. Reported, never scored: no interval or Orbit on this page consults them.
Model	Arm	Records that reasoned	Mean thinking tokens	Most on one record
Claude Fable 5	with skill	20 of 20	609	1,138
Claude Fable 5	no skill	20 of 20	483	931

Fable 5.1, one pass

one pass over each committed item. Matrix hash db373661. Pre-registration · Run report.

One row per model. Every figure is read from this run’s committed cells.
Model	No skill	With skill	Delta	Interval	Pairs	Orbit	Injected tokens	False positives per page
Claude Fable 5.1	0.9833	0.9500	-0.033	-0.099 to 0.032	20	In free drift	not measured	6 with, 5 without

Expansion cohort (three arms)

one pass over each committed item. Matrix hash 0078cfa8. Pre-registration · Run report.

One row per model. Every figure is read from this run’s committed cells.
Model	No skill	With skill	Delta	Interval	Pairs	Orbit	Injected tokens	False positives per page
Claude Fable 5.1	0.9298	0.9833	+0.054	-0.017 to 0.124	10	In free drift	not measured: this run records the dose per arm, in the doses list	5 with, 6 without
Claude Haiku 4.5	0.4345	0.4114	-0.023	-0.214 to 0.168	10	In free drift	not measured: this run records the dose per arm, in the doses list	25 with, 29 without
Claude Opus 5	0.8333	0.7917	-0.042	-0.178 to 0.094	10	In free drift	not measured: this run records the dose per arm, in the doses list	11 with, 8 without
Claude Sonnet 5	0.5538	0.6288	+0.075	-0.030 to 0.180	10	In free drift	not measured: this run records the dose per arm, in the doses list	22 with, 22 without

Every figure above is computed at build time from harness/results/runs-skills.jsonl and harness/results/ledger-skills.jsonl, both committed, by the frozen status_v1 rule and the metrics_v1 cost definitions. How a claim gets tested.