Does Skill creation walkthrough help? Tested on writing skills tasks
Skill creation walkthrough, from rampstackco/claude-skills. Its skill authoring tasks were run with the skill loaded and without it. Each time it is the same task, run twice.
What we tested
Whether loading this skill helps a model write a skill document.
What counts as helping
The score with the skill has to beat the score without it by at least 10 percentage points, on the same 10 briefs.
How we scored it
We checked each document against the open Agent Skills specification and counted the rules it passed. The score is the share it passed, from 0 to 1.
Without to with, per model
Via API tested via API
Claude Haiku 4.5 First run, with and without the skill
Holds
Gemini 3.1 Flash Lite First run, with and without the skill
No measured effect
Gemini 3.1 Flash Lite Wider run, three ways
Holds
GPT-5 mini First run, with and without the skill
Holds
GPT-5 mini Wider run, three ways
Holds
deterministic pass rate, 0 to 1
In Claude Code tested in Claude Code
Claude Fable 5.1 Fable 5.1, single attempt
Could not measure
Claude Fable 5 Second run, every task twice
No measured effect
Claude Fable 5 First run, single attempt
No measured effect
Claude Fable 5.1 Wider run, three ways
No measured effect
Claude Haiku 4.5 Second run, every task twice
Holds
Claude Haiku 4.5 Wider run, three ways
Holds
Claude Opus 5 First run, single attempt
Could not measure
Claude Opus 5 Second run, every task twice
Could not measure
Claude Opus 5 Wider run, three ways
Could not measure
Claude Sonnet 5 Second run, every task twice
No measured effect
Claude Sonnet 5 Wider run, three ways
No measured effect
deterministic pass rate, 0 to 1
Show per-task detail
Every item, without and with, per model
Via API tested via API
- Claude Haiku 4.5
- GPT-5 mini
- GPT-5 mini
- Gemini 3.1 Flash Lite
- Gemini 3.1 Flash Lite
deterministic pass rate, 0 to 1
Hover or focus an item to read its task and every model’s two values.
Task-paired: these item means are what this cell's interval is built from.
The item means behind this plot
| Item | Claude Haiku 4.5 | GPT-5 mini | GPT-5 mini | Gemini 3.1 Flash Lite | Gemini 3.1 Flash Lite | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Task | without | with | without | with | without | with | without | with | without | with |
| sa-05 | 0.545 | 1.000 | 0.273 | 1.000 | 0.273 | 1.000 | 0.545 | 1.000 | 0.909 | 1.000 |
| sa-10 | 0.545 | 1.000 | 0.727 | 1.000 | 0.727 | 1.000 | 0.545 | 1.000 | 0.545 | 1.000 |
| sa-02 | 0.545 | 1.000 | 0.273 | 1.000 | 0.818 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 |
| sa-08 | 0.545 | 1.000 | 0.909 | 1.000 | 0.727 | 1.000 | 0.364 | 1.000 | 0.909 | 1.000 |
| sa-07 | 0.545 | 1.000 | 0.273 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 |
| sa-09 | 0.545 | 1.000 | 0.909 | 1.000 | 0.273 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 |
| sa-06 | 0.545 | 1.000 | 0.727 | 1.000 | 0.909 | 1.000 | 0.545 | 1.000 | 0.909 | 1.000 |
| sa-04 | 0.909 | 1.000 | 0.273 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 |
| sa-01 | 0.545 | 1.000 | 0.909 | 1.000 | 0.727 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 |
| sa-03 | 0.909 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 | 0.909 | 1.000 |
Without to with, per model
In Claude Code tested in Claude Code
Claude Haiku 4.5 Expansion cohort (three arms)
Holds
Claude Haiku 4.5 Panel v2 (repeat sampling)
Holds
Claude Fable 5 Panel v2 (repeat sampling)
No measured effect
Claude Fable 5 Skill cohort (one pass)
No measured effect
Claude Fable 5.1 Expansion cohort (three arms)
No measured effect
Claude Fable 5.1 Fable 5.1, one pass
Could not measure
Claude Opus 5 Expansion cohort (three arms)
Could not measure
Claude Opus 5 Panel v2 (repeat sampling)
Could not measure
Claude Opus 5 Skill cohort (one pass)
Could not measure
Claude Sonnet 5 Expansion cohort (three arms)
No measured effect
Claude Sonnet 5 Panel v2 (repeat sampling)
No measured effect
deterministic pass rate, 0 to 1
The Claude Code runs committed their cells as arm means and intervals rather than as per-item values, so this panel is drawn without to with rather than item by item.
What exactly was tested, and how it was scored
- Repository
- rampstackco/claude-skills
- Path
- skills/skill-creation-walkthrough
- Commit
0479242522549dfdb389bb9b7807ad4d6016ffb7- Content hash
8d539bb7a97786875b4489a9005434c2f7c772c8f009994a5ae1ef2b66e8da80- Date tested
- 2026-08-30
The pin is the whole of this skill's identity here. It resolves at https://github.com/rampstackco/claude-skills/tree/0479242522549dfdb389bb9b7807ad4d6016ffb7/skills/skill-creation-walkthrough, and the content hash is a sha256 over exactly the text the model was given with the skill loaded. Nothing else about the skill appears on this site.
S01-skill-creation-walkthrough
Loading skills/skill-creation-walkthrough from rampstackco/claude-skills improves conformance to the open Agent Skills specification on skill authoring tasks.
- Pass criterion
- With-arm mean conformance to the open Agent Skills specification exceeds the without-arm by at least 0.10 on the same 10 briefs at the same model and settings. Key B, the house standard, is reported beside this figure and takes no part in it.
- Scale
- unit. Fraction of the applicable Agent Skills specification predicates passed, so a [0,1] fraction, which is the unit scale. The 0.2 unit threshold in status_v1 applies unchanged. A predicate for an optional field the document does not offer is not applicable and leaves the denominator, so a document is never credited for a rule it did not engage.
- Task pairs planned per model
- 10
- Notes
- TWO KEYS, ONE RANKED. Key A is the open Agent Skills specification, pinned at harness/artifacts/keys/agentskills-specification. It is published by neither cohort, it is the format both skills in this class claim to teach, and it is the only key that becomes a verdict. Key B is SKILL_AUTHORING.md, the operator's own house standard: it is still computed and shown, under a disclosure naming whose standard it is, and it is excluded from the cohort ranking and from every cross-cohort statement. The earlier version of this class ranked on Key B alone, which made a lift here partly a measure of agreement with the operator's conventions. Neither key is given to either arm, and the task prompt restates no part of either.
Injected context tokens
| Arm | Context characters | Injected context tokens |
|---|---|---|
| without | 0 | 0 |
| with | 37,008 | 8,687 |
The character count is exact: it is the length of the text the with arm is given, and the content hash above is a sha256 over that same text. The token figure is an estimate at 4.26 characters per token, the ratio the phase 1 run measured over 3,120 calls, and it is labelled an estimate until a run reports its own token counts. The without arm is given the identical prompt and nothing else, so its zero is a measurement rather than a missing value.
Verdict per model
| Model | Reading | Orbit | Without -> with | Effect | 95 percent interval | Task pairs | Model version returned |
|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5 | Deterministic, scored against a committed answer key | Stable | 0.6182 -> 1.0000 | +0.3818 | [0.2868, 0.4768] | 10 pairs, 20 of 20 records | claude-haiku-4-5-20251001 |
| Gemini 3.1 Flash Lite | Deterministic, scored against a committed answer key | In free drift unclear. The model scored 87% without it. | 0.8727 -> 1.0000 | +0.1273 | [0.0560, 0.1985] | 10 pairs, 20 of 20 records | gemini-3.1-flash-lite |
| GPT-5 mini | Deterministic, scored against a committed answer key | Stable | 0.7182 -> 1.0000 | +0.2818 | [0.1282, 0.4354] | 10 pairs, 20 of 20 records | gpt-5-mini-2025-08-07 |
| Gemini 3.1 Flash Lite | Deterministic, scored against a committed answer key | Stable | 0.7455 -> 1.0000 | +0.2545 | [0.1196, 0.3895] | 10 pairs, 30 of 30 records | gemini-3.1-flash-lite |
| GPT-5 mini | Deterministic, scored against a committed answer key | Stable (10 of 10 pairs) | 0.6182 -> 1.0000 | +0.3818 | [0.1925, 0.5711] | 10 pairs, 30 of 33 records | gpt-5-mini-2025-08-07 |
- Stable: the treatment arm outscored the control arm by more than the threshold, and the interval excludes zero.
- In free drift: no separation the design can resolve. Not evidence of no effect.
Cost
| Model | Arm | Tasks attempted | Mean input tokens | Mean output tokens | Cost per task | Cost ratio | Grading cost per task pair |
|---|---|---|---|---|---|---|---|
Claude Haiku 4.5 served claude-haiku-4-5-20251001 | without | 10 | 68 | 1,210 | $0.00612 | ||
Claude Haiku 4.5 served claude-haiku-4-5-20251001 | with | 10 | 8,791 | 1,533 | $0.01645 | 2.69x | |
GPT-5 mini served gpt-5-mini-2025-08-07 | without | 10 | 61 | 1,459 | $0.00293 | ||
GPT-5 mini served gpt-5-mini-2025-08-07 | with | 10 | 8,003 | 1,583 | $0.00517 | 1.76x | |
| Gemini 3.1 Flash Lite | without | 10 | 59 | 580 | $0.00088 | ||
| Gemini 3.1 Flash Lite | with | 10 | 8,376 | 713 | $0.00316 | 3.58x |
Defined in the metrics canon. Cost per task divides every dollar spent on an arm by the tasks attempted on it, including tasks whose call returned nothing, because a call that returned nothing was still billed.
In Claude Code
These cells are tested in Claude Code, on a subscription path with no API key. They are a second instrument: no figure here is averaged with one tested via API above. How the two were compared.
- Claude Fable 5 · unclear, and both runs agree
- Claude Opus 5 · could not be measured, and both runs agree
- Claude Haiku 4.5 · helped under repeat sampling
- Claude Sonnet 5 · unclear under repeat sampling
- Claude Fable 5.1 · could not be measured in the version re-test
Skill cohort (one pass)
one pass over each committed item. Matrix hash 2707a06a. No pre-registration: this run predates the practice on this arm. Run report.
| Model | No skill | With skill | Delta | Interval | Pairs | Orbit | Injected tokens |
|---|---|---|---|---|---|---|---|
| Claude Fable 5 | 1.0000 | 0.9273 | -0.073 | -0.215 to 0.070 | 10 | In free drift | 12,321 |
| Claude Opus 5 | 1.0000 | 1.0000 | +0.000 | 0.000 to 0.000 | 10 | Unobservable | 12,321 |
| Model | Arm | Records that reasoned | Mean thinking tokens | Most on one record |
|---|---|---|---|---|
| Claude Fable 5 | with skill | 20 of 20 | 68 | 231 |
| Claude Fable 5 | no skill | 20 of 20 | 64 | 127 |
Panel v2 (repeat sampling)
two passes over each committed item, under repeat sampling. Matrix hash 94bab960. Pre-registration · Run report.
| Model | No skill | With skill | Delta | Interval | Pairs | Orbit | Injected tokens |
|---|---|---|---|---|---|---|---|
| Claude Fable 5 | 1.0000 | 0.9273 | -0.073 | -0.168 to 0.022 | 10 | In free drift | 258,944 |
| Claude Haiku 4.5 | 0.6909 | 1.0000 | +0.309 | 0.220 to 0.398 | 10 | Stable | 183,796 |
| Claude Opus 5 | 1.0000 | 1.0000 | +0.000 | 0.000 to 0.000 | 10 | Unobservable | 258,944 |
| Claude Sonnet 5 | 1.0000 | 0.8909 | -0.109 | -0.218 to -0.000 | 10 | In free drift | 271,984 |
| Model | Arm | Records that reasoned | Mean thinking tokens | Most on one record |
|---|---|---|---|---|
| Claude Fable 5 | with skill | 20 of 20 | 68 | 231 |
| Claude Fable 5 | no skill | 20 of 20 | 64 | 127 |
Fable 5.1, one pass
one pass over each committed item. Matrix hash db373661. Pre-registration · Run report.
| Model | No skill | With skill | Delta | Interval | Pairs | Orbit | Injected tokens |
|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 1.0000 | 1.0000 | +0.000 | 0.000 to 0.000 | 20 | Unobservable | not measured |
Expansion cohort (three arms)
one pass over each committed item. Matrix hash 0078cfa8. Pre-registration · Run report.
| Model | No skill | With skill | Delta | Interval | Pairs | Orbit | Injected tokens |
|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 1.0000 | 0.8545 | -0.145 | -0.336 to 0.045 | 10 | In free drift | not measured: this run records the dose per arm, in the doses list |
| Claude Haiku 4.5 | 0.5455 | 1.0000 | +0.455 | 0.455 to 0.455 | 10 | Stable | not measured: this run records the dose per arm, in the doses list |
| Claude Opus 5 | 1.0000 | 1.0000 | +0.000 | 0.000 to 0.000 | 10 | Unobservable | not measured: this run records the dose per arm, in the doses list |
| Claude Sonnet 5 | 1.0000 | 0.8545 | -0.145 | -0.336 to 0.045 | 10 | In free drift | not measured: this run records the dose per arm, in the doses list |
Key A: the open Agent Skills specification
This class is ranked on conformance to the open specification and on nothing else. The specification is published by neither cohort, it is given to neither arm, and it is pinned like any other artifact: agentskills/agentskills docs at 69ef37e9424c0a7ea9dd2293b559e43ec8176379, content hash b9079c0c10b7930e8c6a20ff2bc10cda2a3343c55185120e3f1116a1a529b220.
| Predicate | What it requires, in our words | Where the specification says it | Scored when |
|---|---|---|---|
frontmatter-present | The document opens with a YAML frontmatter block, and Markdown follows it. | SKILL.md format, opening sentence | always |
frontmatter-name-present | A name field is present. | SKILL.md format, frontmatter table, name row | always |
frontmatter-description-present | A description field is present. | SKILL.md format, frontmatter table, description row | always |
name-within-length-limit | The name is between one and sixty four characters. | name field, first bullet | always |
name-characters-legal | The name uses only lowercase letters, digits and hyphens. | name field, second bullet | always |
name-no-edge-hyphen | The name neither opens nor closes on a hyphen. | name field, third bullet | always |
name-no-consecutive-hyphens | The name carries no run of two hyphens. | name field, fourth bullet | always |
description-within-length-limit | The description is between one and one thousand and twenty four characters. | description field, first bullet | always |
body-present | Instructions follow the frontmatter rather than an empty document. | Body content, opening sentence | always |
within-line-limit | The document stays under the stated line ceiling for a main SKILL.md. | Progressive disclosure, closing line | always |
file-references-one-level-deep | Every relative path the document points at sits at most one directory down. | File references, closing line | always |
compatibility-within-length-limit | If compatibility is given, it stays inside its stated ceiling. | compatibility field, first bullet | the field is offered |
metadata-is-string-map | If metadata is given, it maps string keys to string values. | metadata field, first bullet | the field is offered |
allowed-tools-is-one-string | If allowed-tools is given, it is a single space separated string. | allowed-tools field, first bullet | the field is offered |
A predicate for an optional field a document does not offer is not applicable and leaves the denominator, so a document is never credited for a rule it did not engage.
Key B: the operator's house standard, not ranked
| Claim | Model | Without | With | Task pairs |
|---|---|---|---|---|
| S01-skill-creation-walkthrough | Claude Haiku 4.5 | 0.2583 | 0.4750 | 10 |
| S01-skill-creation-walkthrough | GPT-5 mini | 0.2083 | 0.3750 | 10 |
| S01-skill-creation-walkthrough | Gemini 3.1 Flash Lite | 0.2500 | 0.5000 | 10 |
No interval, no orbit and no verdict appears above, and none can: these are two arm means and the frozen rule is never called on them.
Every figure above is computed at build time from harness/results/runs-skills.jsonl and harness/results/ledger-skills.jsonl, both committed, by the frozen status_v1 rule and the metrics_v1 cost definitions. How a claim gets tested.