How We Test Prompting Tips and Skills

Where claims come from

Claims are advice about prompting that circulates in practitioner communities: forums, documentation, and the working folklore of people who use these models daily. A claim qualifies for testing when it is stated widely enough to matter and specific enough to be wrong.

That a claim circulates is an input signal. It decides what gets tested. It is never evidence about whether the claim holds, and it never appears on a claim page as a number.

Making a claim testable

A claim becomes a test when it can be written as two prompts that differ in exactly one respect. One is the control arm, one is the treatment arm, and the difference between them is the whole of the manipulated variable. Everything else, including the task content, is identical by construction, because both arms are built by substituting the same task into two templates.

This is a paired comparison on a small fixed task set, not a controlled experiment on a representative sample. It supports no causal reading beyond the pairing.

The harness

18 claims ran against 3 models, both arms, on fixed task sets, producing 54 claim and model pairs from 4,938 records. 4 of the launch claims are rubric graded and the rest are scored by committed deterministic functions.

Those records arrived in two runs, on two instruments, and the table below is both of them. The split is set out under two instruments, one rule, because the difference changes how an interval on a wave 2 row was built and a reader comparing two rows should know which is which.

A validity gate runs before scoring. A call that returned nothing usable, whether an empty output or a transport failure, is marked invalid, is never scored, and never enters a denominator. It is not a score of zero. Scoring an empty answer as zero was a real defect in an earlier instrument, and it is described below.

Two instruments, one rule

The claims were not all measured the same way. The launch run and the wave 2 run used different instruments, and rather than hide that behind one table, the difference is stated: it is a fact about how the numbers were made.

What did not change is the rule. Every verdict in the table, from either run, is assigned by the same frozen status_v1 classification, from the same file, with the same thresholds. Only the way an interval is built differs, and it differs because the two runs sampled differently.

Why a wave 2 interval is built differently

The launch run treats each record as one measurement, and its interval widens or narrows with the number of records behind it. That is right when every record is a separate look at the question.

Wave 2 is not that. Its task sets hold ten items, the design asked for fifty observations per cell, and the runner closed the gap by walking the same ten items five times. Every call ran at temperature 0, so the same prompt returned the same answer and scored the same: not five looks at a question, but one look written down five times. Counting those as fifty makes the interval about twice as narrow as the evidence supports, and the width of the interval is exactly what decides whether a verdict is allowed.

So on wave 2 the repeated passes collapse back to their item, the two arms are matched on the items they share, and the interval is built from the per-item differences. The unit is the pair, because the pair is what was independently measured. This is not a refinement made after the fact: it corrected a verdict that had already been published, which was then unpublished. The full account is in harness/PSEUDO-REPLICATION.md.

Coverage is the one figure that stays in records on both runs. It answers how much of what was attempted survived, which is a question about calls, not about observations, so the pairing is kept out of it.

Sets are calibrated before they are funded

A paid probe runs before a wave 2 task set is bought in full. It asks whether the control arm leaves the treatment arm anywhere to go: a set where the control already scores at the top cannot show an improvement, and one at the bottom cannot show a decline. Either way the run would buy a cell that measures nothing.

The control arm has to land between 0.1 and 0.9 of the scale to clear the gate. That is what the wave 2 probe checked, at 30 per set, on claude-haiku-4-5, on 2026-08-17, against a cap of 15 dollars. It cleared: C16-cutoff-disclosure came back at 86.0 percent of the ceiling, inside the walls, with room to move in either direction.

Saturation is the failure this is designed to catch, and the launch run has a worked example of missing it: the original three-example claim scored full marks in both arms on every model, which is not a null result but a task set that was too easy to separate anything. Its record is kept on the page that replaced it.

A claim's own target is not the threshold

Two numbers on this site look alike and are not. Each wave 2 claim carries a passCriterion, which is a design target: how large an effect the task set was built to be able to detect. It is set by the person building the set, and it answers whether the set is worth paying for.

The status_v1 threshold is what decides a verdict. It is fixed per scale, it is the same for every claim, and nothing in a claim file can change it. A claim can clear its own design target and still be In free drift, and that is not a contradiction: it means the set worked and the effect it found was real but smaller than the rule calls decisive.

Reading one as the other cost a re-run. The C14-prompt-chaining cell on gemini-3.1-flash-lite measured +0.100, which read against a design target of the same size looked like a boundary case, and it was authorised for more data. Under the rule it was never near a boundary, because the rule compares it against 0.2 and its verdict is In free drift. The extra records narrowed the interval exactly as more data should, and the verdict never moved. The account is in harness/PASS-CRITERION-VS-THRESHOLD.md.

One claim landed on both sides of the rule: telling a model its knowledge may be out of date cut confident fabrication on gpt-5-mini and gemini-3.1-flash-lite, and increased it on claude-haiku-4-5, which is the whole case for publishing every model separately rather than one verdict per claim.

Claude Code as a second instrument

Some claims on this site were measured twice: once through the Anthropic API, and once through the Claude Code CLI on a subscription. Those are two instruments, not two runs of one, and nothing on this site averages a figure from one with a figure from the other.

`claude -p` runs the Claude Code CLI in print mode: it sends one prompt to a real Claude Code session and returns the answer, without an interactive terminal.

It authenticates against a subscription rather than an API key, which is the whole reason this path exists: it measures what a Claude Code user's requests actually meet.

The session is isolated by flags rather than by `--bare`, because `--bare` forces API-key authentication and would defeat the purpose. Tools, slash commands and MCP servers are disabled, the working directory is outside any repository so no instruction file is discoverable above it, and the system prompt is replaced.

Two divergences from the API path are carried on every record unchanged: the CLI exposes no sampling flag where the API path pinned temperature 0, and the child is a real Claude Code session rather than a bare model call.

What the comparison found

Each comparison was registered before its run and asks one question: does the Claude Code interval overlap the API interval on the same cell? Every figure below is read from the artifact that recorded it.

The single-turn row is the one that fixed the label. One cell of two failed to overlap and changed sign, and the rule registered before that calibration ran said that a single such failure makes the CLI path a second instrument. The other two comparisons found agreement and do not revise it: continuity on a cell is not a promotion of the transport.

A model was substituted, silently, and it is caught now

The plan served a different model than the one requested on 35 of 338 records of the skill run, answering as claude-opus-5. The prompts, the flags and the stop reason were all ordinary and the answers scored plausibly. Nothing in the envelope announced it except the served model string.

2 cells have their verdict withheld because of it, published as withheld with the number of task pairs that survived rather than as a thinner measurement of the model that was asked for.

• The served model is read back from the CLI's own usage envelope and recorded on every record, so a substitution is visible in the committed data rather than inferred from a score that looks wrong.
• A record whose served model differs from the requested one is invalid at birth and invalid at load. Both, because a mark written at birth can be lost or edited on disk, and a check applied only at load would let an unmarked record past any other reader.
• Its tokens still count against the run's spend. The call was made and consumed plan usage whoever served it, so deleting it would understate what the run cost.
• A cell left with too few surviving pairs is withheld with its reason and its remaining pair count, never published as a thinner measurement of the model that was asked for.

What the compute was, and what was not read

Every Claude Code call on this site was made against a Max subscription. No API key reaches the child process and nothing on this path is billed, so the honest cost noun is plan usage per run: the share of a subscription window a run consumed, read by a person from a vendor page before and after. Nothing in the harness can observe it, which is why these are slots a person fills.

The two percentages are not a difference. The bracket crossed the plan's weekly reset, so the two percentages are measured against different allowances and their difference means nothing. The window also covers the skill cohort run, both tips cohorts and ordinary interactive Claude Code use. The readings are published as observations and are informational: no figure on this site is derived from them.

The console check is closed. A console balance and spend figure that moved across one of these brackets would mean something in the window went out over an API key, which would be a finding about the run rather than a cost of it. The figure was read at both edges of the bracket and did not move, so the zero delta is an observation. This is the independent evidence that nothing in the window was billed over an API key, and it does not rest on the absence of a key alone.

The structural argument stands beside the reading rather than in place of it: no API key reaches the child, and the gate proofs say so before every run. What changed is that the claim no longer rests on that argument alone.

Codex as a third instrument

Two claims on this site were measured a third way: through the Codex command line, on a ChatGPT subscription. That is a third instrument, not a third run of one, and nothing on this site averages a figure from it with a figure from either of the others.

The label these results carry, in full, is tested in Codex (replaced base instructions; runtime tools, permissions and system skills present). The parenthesis is not a hedge, it is the specification: the model ran with its base instructions replaced by ours, and with the Codex runtime’s own tools, permission model and system skills still present. A reader setting a Codex figure beside an API figure is comparing two different amounts of scaffolding and is entitled to know it.

Scoped to one build, read back from the answers rather than declared: codex-cli 0.153.4.

One limitation, dated 2026-09-07: built-in app tools were reachable; no record invoked one; disabled from this version on. Reachable and invoked are different claims. All 358 committed records were scanned on both streams and none carries an MCP tool call, so the cells stand.

What the 3 runs published, and what they withheld

Across those runs, 34 claims measured every planned unit and are published. The rest were withheld rather than published short, each for its own reason.

C16-cutoff-disclosure has no cell on GPT-5.4 mini. That run measured 38 of its 60 planned units and is withheld rather than published short: model no longer served to this account after 38 of 60 units; two retries refused. Its records are retained unchanged.

C16-cutoff-disclosure has no cell on GPT-5.6 Luna. That run measured 54 of its 60 planned units and is withheld rather than published short: 6 units exhausted the single C8 retry after policy-blocked built-in tool attempts; no completed tool or MCP hit. Full one-pass coverage not reached.. Its records are retained unchanged.

What those runs met beyond their own numbers is recorded with each run rather than here. Those are facts about one model on one day, so they are published on that run’s own results and are not carried onto the instrument.

The standing caveat: which models a ChatGPT account is served on this transport changes without notice: the set is not announced, not versioned and not readable from the CLI, so a model that answered on one call can be refused on the next with no change to the build, the account or the request, and the only way to know what is served is to ask and record the answer with a date. So a Codex figure on this site is a reading on a model nobody can promise is still served. What was served on a given day was measured and written down, in the availability recon of 2026-09-08.

The one line every call carried

Codex refuses to run with an empty instructions file, so there was no wording that made the two paths identical: the choice was between one line and no Codex results at all. Every call on this instrument carried this sentence, the same on both sides of every comparison:

Answer the user's message.

The comparison inside each pair survives it. Whatever that sentence does to an answer, it does to both sides equally, so the difference we measure still isolates the thing being tested, and every Codex result is built the same way as every other result on this site. The comparison between test methods is what pays for it. The results tested via API carried nothing in that position, so a Codex figure set beside one of those differs by that line as well as by the tip being tested, and only the second is what the tip is about.

Which model answered: there is no answer here

This test method returns no name for the model that produced an answer. We looked for one twice, two different ways, once through the same code that would go on to write the results; it is not in the trace and not in the reply. So every answer carries this instead, and a test asserts it is on all of them:

served identity: requested model accepted by the endpoint; no served string available on this instrument

Wherever this site would print the name of the model that answered, it prints “served model not reported by this instrument” instead. What replaces the check is weaker than the check, and is written down as weaker. A request for a model the account may not use is refused outright rather than quietly answered by something else, which is evidence that this path refuses rather than swaps. It is not evidence about a request that succeeded. On the one path that does return a name, we found 71 answers that came from a different model than the one we asked for, and we found them only because the name was there to compare against.

No Codex figure on this site carries a cost, not even as an estimate. The other subscription path prices its answers at the published rate for the model that served them; this one has no served model to price, so a figure here would be an unmeasured quantity converted at a rate we do not know applies.

What the comparison found

It was registered before the run and asks one question: does the Codex bracket overlap the bracket from the same model tested through the API, on the same test? A second set of 356 calls was bought through the API to answer it, on the same model over the same tasks, because a comparison against a different model would not have been one.

continuous with the API twin, 2 of 2 cells overlap. Two single-turn claims, replayed on one model against a same-model API twin bought for the comparison. Read from the file that recorded it.

The status_v1 rule

Every verdict on this site is assigned by one frozen rule. The site cannot vary it per claim, per model, or per page. A cell is one claim against one model, over both arms.

The comparison is the difference between the treatment mean and the control mean, with a 95 percent interval on that difference. A verdict requires the interval to exclude zero and the difference to clear a threshold.

These thresholds are not chosen by taste. Each is the widest interval half-width the design produced on that scale, rounded up to a 0.05 step, over cells where at least one arm varied. A difference can only clear zero when it exceeds its own half-width, so the widest half-width observed is the smallest gap the design can be trusted to resolve. On the 0 to 1 scale most cells had no variance in either arm and could not contribute a half-width, so that threshold rests on a thin basis and is published as one.

Comparisons carry a fixed slack of 1e-9. Means are sums of floating point numbers, so a difference sitting exactly on a threshold can otherwise land either side of it in the sixteenth decimal place. The slack is far below the finest distinction the scorers can produce, so it cannot change a verdict the data supports.

The Orbit values

Unobservable is tested before the thresholds. It covers two cases: an unbounded metric, where a fixed gap is not comparable to a gap on a bounded scorer and would be a number without a meaning; and both arms sitting at the same bound of the scale, where the task set did not separate the conditions. The second case is a fact about the instrument, not a measured null. Publishing it as In free drift would tell a reader the site measured no effect when it measured nothing at all.

A cell whose arms are both constant but at different bounds is not this case. Those cells are decisive, and they are classified normally.

Decaying requires the same cell measured on two runs. The record set is a single run, so no cell has a prior verdict to fall from and the value is unreachable today. It is defined so the rule does not have to change when re-testing begins, and the engine asserts in test that it is never emitted.

Coverage

Coverage is valid records over records attempted, for a cell, across both arms. It is computed separately from the verdict and never affects it. Where the two figures differ, both are shown, so a shortfall is visible rather than hidden behind a clean denominator.

How the lead claims are chosen

The lead is C06-permit-idk, the one claim that is Stable on every model in the set. That is a stated editorial constant, recorded so the choice is auditable rather than implicit.

The co-lead is computed, not chosen. It is the most-sampled resolved null: of every cell whose verdict is In free drift, keep those with complete coverage, rank by the number of valid records behind the null, and break ties on the widest interval. The intent is to surface the null the pilot looked hardest at and still could not resolve into an effect.

On the current records that selects C05-think-step-by-step on gemini-3.1-flash-lite, with 100 valid records behind a verdict of In free drift. It was selected from 10 clean nulls by that ranking, with no human input at any step.

Computing this rather than choosing it is the guarantee that the site does not cherry-pick its wins. The rule is designed to surface the sharpest available statement against the site's own thesis.

Limits of what this measured

Grading is the expense, not generation

The launch run's grader accounted for 64.6 percent of that run's spend, 4.4599 of 6.9071 dollars, across 710 launch verdicts; wave 2 ran deterministically with no grader calls. A rubric graded record cost about 19.9 times a deterministic one. This constrains how much rubric scored work the project can afford, and therefore how many claims can be tested that way.

The reasoning budget was a confound, and fixing it changed results

On an earlier instrument, with a tighter output budget, 11.6 percent of one vendor's calls spent the whole budget reasoning and returned empty text: 121 of 1040. Those were scored as zero, which looked like data, and the failure fell about 2.1 times as often on the treatment arm as the control arm, 82 against 39, because treatment prompts are longer. That correlated the manipulated variable with the failure and reversed the direction of every graded result for that vendor. With the budget raised, the empty-output residue is 0 across all 3 vendors. No figure from the earlier instrument is comparable to a figure published here.

There is no parity across vendors

One vendor rejects a temperature of 0 and ran at its own default on every call, with the rejection text recorded on each record. The others ran at 0. Reading one model column against another therefore compares two settings as well as two models. Separately, one vendor reports no thinking token figure at all, so an accepted suppression setting is consistent with suppression but does not prove it.

Model selection was a measurement decision

The pilot needed models that accept fixed sampling and do not reason by default, because a model that reasons before answering gives the control arm the behaviour the treatment arm asks for. Two current models from one vendor fail that test and were not used. The verdicts belong to the versions recorded on the records, not to any vendor's current best model.

Coverage is not uniform

One cell lost ten records to a transport failure across a 95 second window. One record carries a valid answer whose grader verdict could not be parsed, and is in neither the valid count nor the invalid count, so that cell's columns do not sum to its arm total. Both are named on the claim pages that carry them.

Saturation

11 of the 54 cells have both arms at a bound of the scale. Those task sets did not separate the conditions, and most of the claims that were deliberately hardened after an earlier run remained saturated afterwards. The hardening did not work. That is reported as a finding about the claims and the task design rather than smoothed into an inconclusive interval.

Corrections

Every correction to a published figure, in date order, appended and never rewritten. Where a correction is about one claim it also appears on that claim’s page; where it is about a set of pages it is published here once and linked from each of them.

What is not published

The method publishes in full. The operational mechanics of how claims are captured, and the systems that run the pipeline, are not part of the method and do not publish.