Does Code review web help? Tested on code review tasks
Code review web, from rampstackco/claude-skills. Its code review tasks were run with the skill loaded and without it. Each time it is the same task, run twice.
What we tested
Whether loading this skill helps a model review code.
What counts as helping
The skill has to find at least 10 percentage points more of the planted bugs than the model finds without it, on the same 10 source files.
How we scored it
We planted known bugs across the 10 source files and counted how many the model named. The score is the share it found, from 0 to 1.
Without to with, per model
Via API tested via API
GPT-5 mini
No measured effect
Gemini 3.1 Flash Lite
No measured effect
deterministic pass rate, 0 to 1
In Claude Code tested in Claude Code
Claude Opus 5 Wider run, three ways
No measured effect
Claude Fable 5.1 Wider run, three ways
No measured effect
Claude Sonnet 5 Wider run, three ways
No measured effect
Claude Haiku 4.5 Wider run, three ways
No measured effect
deterministic pass rate, 0 to 1
Show per-task detail
Every item, without and with, per model
Via API tested via API
- Gemini 3.1 Flash Lite
- GPT-5 mini
deterministic pass rate, 0 to 1
Hover or focus an item to read its task and every model’s two values.
Task-paired: these item means are what this cell's interval is built from.
The item means behind this plot
| Item | Gemini 3.1 Flash Lite | GPT-5 mini | ||
|---|---|---|---|---|
| Task | without | with | without | with |
| c10 | 0.250 | 0.500 | 0.500 | 0.750 |
| c02 | 0.400 | 0.600 | 0.600 | 0.600 |
| c04 | 0.400 | 0.000 | 0.600 | 0.400 |
| c05 | 0.500 | 0.333 | 0.500 | 0.500 |
| c07 | 0.400 | 0.000 | 0.600 | 0.600 |
| c06 | 0.667 | 0.333 | 0.667 | 0.667 |
| c08 | 0.667 | 0.500 | 0.833 | 0.667 |
| c01 | 0.800 | 0.600 | 0.800 | 1.000 |
| c03 | 0.833 | 0.667 | 1.000 | 0.833 |
| c09 | 1.000 | 1.000 | 1.000 | 1.000 |
Without to with, per model
In Claude Code tested in Claude Code
Claude Haiku 4.5 Expansion cohort (three arms)
No measured effect
Claude Sonnet 5 Expansion cohort (three arms)
No measured effect
Claude Fable 5.1 Expansion cohort (three arms)
No measured effect
Claude Opus 5 Expansion cohort (three arms)
No measured effect
deterministic pass rate, 0 to 1
The Claude Code runs committed their cells as arm means and intervals rather than as per-item values, so this panel is drawn without to with rather than item by item.
What exactly was tested, and how it was scored
- Repository
- rampstackco/claude-skills
- Path
- skills/code-review-web
- Commit
a67dd34c609f034c0cfd736a348659bbdf1605bf- Content hash
a27bba1351840ced78f98f02c94be30140cb159d940f866455ef0ca60c6183d5- Date tested
- not yet tested
The pin is the whole of this skill's identity here. It resolves at https://github.com/rampstackco/claude-skills/tree/a67dd34c609f034c0cfd736a348659bbdf1605bf/skills/code-review-web, and the content hash is a sha256 over exactly the text the model was given with the skill loaded. Nothing else about the skill appears on this site.
S11-code-review-web
Loading skills/code-review-web from rampstackco/claude-skills improves outputs on code review tasks.
- Pass criterion
- With-arm mean coverage exceeds the without-arm by at least 0.10 on the same 10 fixture files at the same model and settings.
- Scale
- unit. Defects found over defects seeded, so a [0,1] fraction, which is the unit scale. The same scorer, the same scoring mode and the same scale as the accessibility and on-page audit classes.
- Task pairs planned per model
- 10
- Notes
- Ten fixture source files across TypeScript, JavaScript, Python, SQL, a Next.js Route Handler and a Supabase client module, each seeded with known defects drawn from a closed list of codes that is given to BOTH arms in the prompt, so the code list is not the manipulated variable. The ledger is at fixtures/code-review/defect-ledger.json and carries a predicate per entry, evaluated against the fixture bytes by the answer key test; entries no predicate decides are marked judgement and listed as unverified. The model is asked for the SYMBOL a defect sits on and never for a line number. False positives are counted on the record and never netted off coverage. The external slot for this class is unfilled and is recorded as unfilled in skill-cohort.json, so the class publishes one row rather than a pair until that selection is made.
Injected context tokens
| Arm | Context characters | Injected context tokens |
|---|---|---|
| without | 0 | 0 |
| with | 31,160 | 7,315 |
The character count is exact: it is the length of the text the with arm is given, and the content hash above is a sha256 over that same text. The token figure is an estimate at 4.26 characters per token, the ratio the phase 1 run measured over 3,120 calls, and it is labelled an estimate until a run reports its own token counts. The without arm is given the identical prompt and nothing else, so its zero is a measurement rather than a missing value.
Verdict per model
| Model | Reading | Orbit | Without -> with | Effect | 95 percent interval | Task pairs | Model version returned |
|---|---|---|---|---|---|---|---|
| Gemini 3.1 Flash Lite | Deterministic, scored against a committed answer key | In free drift unclear. The model scored 57% without it. | 0.5917 -> 0.4533 | -0.1383 | [-0.2793, 0.0026] | 10 pairs, 30 of 30 records | gemini-3.1-flash-lite |
| GPT-5 mini | Deterministic, scored against a committed answer key | In free drift unclear. The model scored 71% without it. | 0.7100 -> 0.7017 | -0.0083 | [-0.1001, 0.0835] | 10 pairs, 30 of 30 records | gpt-5-mini-2025-08-07 |
- In free drift: no separation the design can resolve. Not evidence of no effect.
Cost
Defined in the metrics canon. Cost per task divides every dollar spent on an arm by the tasks attempted on it, including tasks whose call returned nothing, because a call that returned nothing was still billed.
Where the with arm did not win
Gemini 3.1 Flash Lite, Deterministic, scored against a committed answer key
Lost on 7 of 10 pairs: c04 (-0.4000), c07 (-0.4000), c06 (-0.3333), c01 (-0.2000), c03 (-0.1667), c05 (-0.1667), c08 (-0.1667).
Drew on 1 of 10 pairs: c09.
Findings not in the answer key: 13 with the skill, 7 without. These are counted and never netted off coverage.
GPT-5 mini, Deterministic, scored against a committed answer key
Lost on 3 of 10 pairs: c04 (-0.2000), c08 (-0.1667), c03 (-0.1667).
Drew on 5 of 10 pairs: c02, c05, c06, c07, c09.
Findings not in the answer key: 40 with the skill, 45 without. These are counted and never netted off coverage.
In Claude Code
These cells are tested in Claude Code, on a subscription path with no API key. They are a second instrument: no figure here is averaged with one tested via API above. How the two were compared.
- Claude Fable 5.1 · unclear in the three-arm cohort
- Claude Haiku 4.5 · unclear in the three-arm cohort
- Claude Opus 5 · unclear in the three-arm cohort
- Claude Sonnet 5 · unclear in the three-arm cohort
Expansion cohort (three arms)
one pass over each committed item. Matrix hash 0078cfa8. Pre-registration · Run report.
| Model | No skill | With skill | Delta | Interval | Pairs | Orbit | Injected tokens | False positives per page |
|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 0.9017 | 0.9017 | +0.000 | 0.000 to 0.000 | 10 | In free drift | not measured: this run records the dose per arm, in the doses list | 12 with, 7 without |
| Claude Haiku 4.5 | 0.4983 | 0.5167 | +0.018 | -0.169 to 0.206 | 10 | In free drift | not measured: this run records the dose per arm, in the doses list | 12 with, 9 without |
| Claude Opus 5 | 0.9350 | 0.9350 | +0.000 | 0.000 to 0.000 | 10 | In free drift | not measured: this run records the dose per arm, in the doses list | 23 with, 29 without |
| Claude Sonnet 5 | 0.7933 | 0.8333 | +0.040 | -0.012 to 0.092 | 10 | In free drift | not measured: this run records the dose per arm, in the doses list | 12 with, 9 without |
Every figure above is computed at build time from harness/results/runs-skills.jsonl and harness/results/ledger-skills.jsonl, both committed, by the frozen status_v1 rule and the metrics_v1 cost definitions. How a claim gets tested.