Does SEO help? Tested on SEO tasks

SEO, from affaan-m/ECC. Its on-page audit tasks were run with the skill loaded and without it. Each time it is the same task, run twice.

What we tested

Whether loading this skill helps a model do on-page SEO audits.

What counts as helping

The skill has to find at least 10 percentage points more of the planted problems than the model finds without it, on the same 10 test pages.

How we scored it

We planted 43 known problems across the 10 test pages and counted how many the model named. The score is the share it found, from 0 to 1.

Without to with, per model

Via API tested via API

Gemini 3.1 Flash Lite First run, with and without the skill

0.488

0.610

No measured effect

Gemini 3.1 Flash Lite Wider run, three ways

0.556

0.610

No measured effect

GPT-5 mini Wider run, three ways

0.561

0.503

No measured effect

GPT-5 mini First run, with and without the skill

0.448

0.480

No measured effect

Claude Haiku 4.5 First run, with and without the skill

0.239 without and with

No measured effect

deterministic pass rate, 0 to 1

In Claude Code tested in Claude Code

Claude Fable 5.1 Fable 5.1, single attempt

0.983

0.950

No measured effect

Claude Fable 5.1 Wider run, three ways

0.888

0.950

No measured effect

Claude Fable 5 Second run, every task twice

0.867

0.940

No measured effect

Claude Fable 5 First run, single attempt

0.942

0.933

No measured effect

Claude Opus 5 Second run, every task twice

0.892

0.832

No measured effect

Claude Opus 5 First run, single attempt

0.867

0.805

No measured effect

Claude Opus 5 Wider run, three ways

0.867

0.788

No measured effect

Claude Sonnet 5 Wider run, three ways

0.601

0.665

No measured effect

Claude Sonnet 5 Second run, every task twice

0.594

0.647

No measured effect

Claude Haiku 4.5 Wider run, three ways

0.343

0.363

No measured effect

Claude Haiku 4.5 Second run, every task twice

0.214

0.323

No measured effect

deterministic pass rate, 0 to 1

A picture of the per-model numbers, drawn from the same results. The tables are the source. Grey is the score without. Colour is the score with. The bracket shows how much the difference could move if we ran it again. Rows are ordered by the score with. A model measured at two versions keeps its versions next to each other. The two test methods are reported separately and never averaged. Where the two scores are the same, the number is printed once at the end of the pair. A bracket wider than the axis is drawn to the edge with its cap omitted; the table gives its bounds.

Show per-task detail

Every item, without and with, per model

Via API tested via API

Claude Haiku 4.5
GPT-5 mini
Gemini 3.1 Flash Lite
Gemini 3.1 Flash Lite
GPT-5 mini

deterministic pass rate, 0 to 1

Hover or focus an item to read its task and every model’s two values.

Task-paired: these item means are what this cell's interval is built from.

The item means behind this plot

Per-item arm means. Without is the unaided arm, with is the treated arm.
Item	Claude Haiku 4.5		GPT-5 mini		Gemini 3.1 Flash Lite		Gemini 3.1 Flash Lite		GPT-5 mini
Task	without	with	without	with	without	with	without	with	without	with
p08	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
p05	0.000	0.000	0.000	0.333	0.333	0.667	0.333	0.667	0.667	0.333
p07	0.000	0.000	0.000	0.250	0.500	0.500	0.750	0.500	0.500	0.500
p02	0.000	0.000	0.600	0.600	0.000	0.800	0.600	0.800	0.600	0.600
p06	0.000	0.000	0.500	0.500	0.500	0.500	0.500	0.500	0.500	0.500
p04	0.333	0.333	0.500	0.667	0.500	0.667	0.333	0.667	0.500	0.500
p10	0.167	0.667	0.667	0.667	0.333	0.500	0.333	0.500	0.667	0.667
p01	0.143	0.143	0.714	0.286	0.714	0.714	0.714	0.714	0.429	0.429
p03	0.750	0.750	1.000	0.500	1.000	0.750	1.000	0.750	0.750	0.500
p09	1.000	0.500	0.500	1.000	1.000	1.000	1.000	1.000	1.000	1.000

A picture of the per-item means behind the per-model numbers. The tables are the source. Each model draws two lines over the same task set: a dashed line through its unaided scores and a solid line through its treated ones. Items are ordered by the mean unaided score across the models that measured them, lowest first. The two instruments are reported separately and never averaged.

Without to with, per model

In Claude Code tested in Claude Code

Claude Haiku 4.5 Panel v2 (repeat sampling)

No measured effect

Claude Haiku 4.5 Expansion cohort (three arms)

No measured effect

Claude Sonnet 5 Panel v2 (repeat sampling)

No measured effect

Claude Sonnet 5 Expansion cohort (three arms)

No measured effect

Claude Opus 5 Expansion cohort (three arms)

No measured effect

Claude Opus 5 Skill cohort (one pass)

No measured effect

Claude Fable 5 Panel v2 (repeat sampling)

No measured effect

Claude Fable 5.1 Expansion cohort (three arms)

No measured effect

Claude Opus 5 Panel v2 (repeat sampling)

No measured effect

Claude Fable 5 Skill cohort (one pass)

No measured effect

Claude Fable 5.1 Fable 5.1, one pass

No measured effect

deterministic pass rate, 0 to 1

A picture of the per-model numbers, drawn from the same results. The tables are the source. Each row runs from the score without to the score with. The thin bar beneath it is the bracket, and it shows how much the difference could move if we ran it again. The two test methods are reported separately and never averaged. A bracket wider than the axis is drawn to the edge with its end cap omitted; the table gives its bounds.

The Claude Code runs committed their cells as arm means and intervals rather than as per-item values, so this panel is drawn without to with rather than item by item.

What exactly was tested, and how it was scored

Repository: affaan-m/ECC
Path: skills/seo
Commit: d8409a4b0813771235555e32e3d8046a73988bfa
Content hash: 9a655a52cfd9f99b4db6226e9e648eea25c91909e05722ad772aee10fb755793
Date tested: 2026-08-30

The pin is the whole of this skill's identity here. It resolves at https://github.com/affaan-m/ECC/tree/d8409a4b0813771235555e32e3d8046a73988bfa/skills/seo, and the content hash is a sha256 over exactly the text the model was given with the skill loaded. Nothing else about the skill appears on this site.

`S08-ecc-seo`

Loading skills/seo from affaan-m/ECC improves outputs on on-page audit tasks.

Pass criterion: With-arm mean coverage exceeds the without-arm by at least 0.10 on the same 10 fixture pages at the same model and settings.
Scale: unit. Same scorer, same ledger, same scale as S07.
Task pairs planned per model: 10
Notes: Shares the on-page-audit task set with S07.

Injected context tokens

Injected context tokens, per arm
Arm	Context characters	Injected context tokens
without	0	0
with	4,337	1,018

The character count is exact: it is the length of the text the with arm is given, and the content hash above is a sha256 over that same text. The token figure is an estimate at 4.26 characters per token, the ratio the phase 1 run measured over 3,120 calls, and it is labelled an estimate until a run reports its own token counts. The without arm is given the identical prompt and nothing else, so its zero is a measurement rather than a missing value.

Verdict per model

Effect per model
Model	Reading	Orbit	Without -> with	Effect	95 percent interval	Task pairs	Model version returned
Claude Haiku 4.5	Deterministic, scored against a committed answer key	In free drift unclear. The model scored 24% without it.	0.2393 -> 0.2393	+0.0000	[-0.1461, 0.1461]	10 pairs, 20 of 20 records	`claude-haiku-4-5-20251001`
Gemini 3.1 Flash Lite	Deterministic, scored against a committed answer key	In free drift unclear. The model scored 49% without it.	0.4881 -> 0.6098	+0.1217	[-0.0538, 0.2971]	10 pairs, 20 of 20 records	`gemini-3.1-flash-lite`
GPT-5 mini	Deterministic, scored against a committed answer key	In free drift unclear. The model scored 45% without it.	0.4481 -> 0.4802	+0.0321	[-0.1611, 0.2254]	10 pairs, 20 of 20 records	`gpt-5-mini-2025-08-07`
Gemini 3.1 Flash Lite	Deterministic, scored against a committed answer key	In free drift unclear. The model scored 49% without it.	0.5564 -> 0.6098	+0.0533	[-0.0751, 0.1818]	10 pairs, 30 of 30 records	`gemini-3.1-flash-lite`
GPT-5 mini	Deterministic, scored against a committed answer key	In free drift unclear. The model scored 54% without it.	0.5612 -> 0.5029	-0.0583	[-0.1355, 0.0189]	10 pairs, 30 of 30 records	`gpt-5-mini-2025-08-07`

In free drift: no separation the design can resolve. Not evidence of no effect.

Cost

Cost per task
Model	Arm	Tasks attempted	Mean input tokens	Mean output tokens	Cost per task	Cost ratio
Claude Haiku 4.5 served `claude-haiku-4-5-20251001`	without	10	667	37	$0.00085
Claude Haiku 4.5 served `claude-haiku-4-5-20251001`	with	10	1,755	54	$0.00203	2.37x
GPT-5 mini served `gpt-5-mini-2025-08-07`	without	10	566	76	$0.00029
GPT-5 mini served `gpt-5-mini-2025-08-07`	with	10	1,531	82	$0.00055	1.86x
Gemini 3.1 Flash Lite	without	10	599	53	$0.00023
Gemini 3.1 Flash Lite	with	10	1,626	52	$0.00048	2.11x

Defined in the metrics canon. Cost per task divides every dollar spent on an arm by the tasks attempted on it, including tasks whose call returned nothing, because a call that returned nothing was still billed.

Where the with arm did not win

Claude Haiku 4.5, Deterministic, scored against a committed answer key

Lost on 1 of 10 pairs: p09 (-0.5000).

Drew on 8 of 10 pairs: p01, p02, p03, p04, p05, p06, p07, p08.

Findings not in the answer key: 36 with the skill, 17 without. These are counted and never netted off coverage.

Gemini 3.1 Flash Lite, Deterministic, scored against a committed answer key

Lost on 1 of 10 pairs: p03 (-0.2500).

Drew on 5 of 10 pairs: p01, p06, p07, p08, p09.

Findings not in the answer key: 19 with the skill, 26 without. These are counted and never netted off coverage.

GPT-5 mini, Deterministic, scored against a committed answer key

Lost on 2 of 10 pairs: p03 (-0.5000), p01 (-0.4286).

Drew on 4 of 10 pairs: p02, p06, p08, p10.

Findings not in the answer key: 63 with the skill, 55 without. These are counted and never netted off coverage.

Gemini 3.1 Flash Lite, Deterministic, scored against a committed answer key

Lost on 2 of 10 pairs: p03 (-0.2500), p07 (-0.2500).

Drew on 4 of 10 pairs: p01, p06, p08, p09.

Findings not in the answer key: 19 with the skill, 26 without. These are counted and never netted off coverage.

GPT-5 mini, Deterministic, scored against a committed answer key

Lost on 2 of 10 pairs: p05 (-0.3333), p03 (-0.2500).

Drew on 8 of 10 pairs: p01, p02, p04, p06, p07, p08, p09, p10.

Findings not in the answer key: 50 with the skill, 44 without. These are counted and never netted off coverage.

In Claude Code

These cells are tested in Claude Code, on a subscription path with no API key. They are a second instrument: no figure here is averaged with one tested via API above. How the two were compared.

Claude Fable 5 · unclear, and both runs agree
Claude Opus 5 · unclear, and both runs agree
Claude Haiku 4.5 · unclear under repeat sampling
Claude Sonnet 5 · unclear under repeat sampling
Claude Fable 5.1 · unclear in the version re-test

Skill cohort (one pass)

one pass over each committed item. Matrix hash 2707a06a. No pre-registration: this run predates the practice on this arm. Run report.

One row per model. Every figure is read from this run’s committed cells.
Model	No skill	With skill	Delta	Interval	Pairs	Orbit	Injected tokens	False positives per page
Claude Fable 5	0.9417	0.9333	-0.008	-0.094 to 0.078	10	In free drift	1,543	7 with, 7 without
Claude Opus 5	0.8667	0.8048	-0.062	-0.143 to 0.019	10	In free drift	1,543	13 with, 8 without

Thinking tokens, recorded per arm. Reported, never scored: no interval or Orbit on this page consults them.
Model	Arm	Records that reasoned	Mean thinking tokens	Most on one record
Claude Fable 5	with skill	20 of 20	609	1,138
Claude Fable 5	no skill	20 of 20	483	931

Panel v2 (repeat sampling)

two passes over each committed item, under repeat sampling. Matrix hash 94bab960. Pre-registration · Run report.

One row per model. Every figure is read from this run’s committed cells.
Model	No skill	With skill	Delta	Interval	Pairs	Orbit	Injected tokens
Claude Fable 5	0.8667	0.9399	+0.073	-0.015 to 0.162	10	In free drift	not measured: 40 of 40 records predate the usage-capture amendment
Claude Haiku 4.5	0.2143	0.3226	+0.108	-0.040 to 0.257	10	In free drift	not measured: 40 of 40 records predate the usage-capture amendment
Claude Opus 5	0.8917	0.8315	-0.060	-0.161 to 0.040	10	In free drift	not measured: 40 of 40 records predate the usage-capture amendment
Claude Sonnet 5	0.5939	0.6471	+0.053	-0.029 to 0.136	10	In free drift	not measured: 40 of 40 records predate the usage-capture amendment

Thinking tokens, recorded per arm. Reported, never scored: no interval or Orbit on this page consults them.
Model	Arm	Records that reasoned	Mean thinking tokens	Most on one record
Claude Fable 5	with skill	20 of 20	609	1,138
Claude Fable 5	no skill	20 of 20	483	931

Fable 5.1, one pass

one pass over each committed item. Matrix hash db373661. Pre-registration · Run report.

One row per model. Every figure is read from this run’s committed cells.
Model	No skill	With skill	Delta	Interval	Pairs	Orbit	Injected tokens	False positives per page
Claude Fable 5.1	0.9833	0.9500	-0.033	-0.099 to 0.032	20	In free drift	not measured	6 with, 5 without

Expansion cohort (three arms)

one pass over each committed item. Matrix hash 0078cfa8. Pre-registration · Run report.

One row per model. Every figure is read from this run’s committed cells.
Model	No skill	With skill	Delta	Interval	Pairs	Orbit	Injected tokens	False positives per page
Claude Fable 5.1	0.8881	0.9500	+0.062	-0.074 to 0.198	10	In free drift	not measured: this run records the dose per arm, in the doses list	6 with, 6 without
Claude Haiku 4.5	0.3429	0.3626	+0.020	-0.203 to 0.242	10	In free drift	not measured: this run records the dose per arm, in the doses list	26 with, 23 without
Claude Opus 5	0.8667	0.7881	-0.079	-0.161 to 0.004	10	In free drift	not measured: this run records the dose per arm, in the doses list	13 with, 8 without
Claude Sonnet 5	0.6014	0.6655	+0.064	-0.108 to 0.236	10	In free drift	not measured: this run records the dose per arm, in the doses list	18 with, 26 without

Every figure above is computed at build time from harness/results/runs-skills.jsonl and harness/results/ledger-skills.jsonl, both committed, by the frozen status_v1 rule and the metrics_v1 cost definitions. How a claim gets tested.