{ "surface": "openaddict-job", "version": "job-rice-v1", "status": "published", "statusMeans": "Published. The figures are from the main runs and the skill runs of this job, read from the committed results file.", "url": "https://openaddict.com/jobs/rice-prioritization", "data": "https://openaddict.com/jobs/rice-prioritization.json", "job": "rice-prioritization", "jobLabel": "RICE prioritization", "title": "RICE Framework with AI: How Accurate Is the Math?", "instruments": [ { "instrument": "claude-code", "label": "tested in Claude Code", "models": [ "claude-opus-5", "claude-sonnet-5", "claude-haiku-4-5", "claude-fable-5-1", "claude-opus-5-5" ] }, { "instrument": "api", "label": "tested via API", "models": [ "gpt-5-mini", "gemini-3.1-flash-lite" ] } ], "rule": "Every figure is read inside one job and one test method. Nothing is averaged across test methods, and a model is ranked only against models given the same backlog size. An absence is a reason, never a zero.", "lastRun": { "date": "2026-10-11", "note": "The newest answer's finish time in harness/results/pm-wave-1/stage-1a-i-main/records.jsonl." }, "answer": [ "Five of the seven models got the RICE arithmetic right on at least nine rows in ten with no help. They were Claude Opus 5 in Claude Code, Claude Sonnet 5 in Claude Code, Claude Fable 5.1 in Claude Code, Claude Opus 5.5 in Claude Code and Gemini 3.1 Flash Lite through the API.", "Claude Haiku 4.5 in Claude Code got 7 of 22 rows right on a 25-row backlog. GPT-5 mini through the API got 7 of 12 rows right on a 15-row backlog.", "The differences are in catching bad data.", "Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Claude Fable 5.1, Claude Opus 5.5 and Gemini 3.1 Flash Lite caught the missing confidence and the odd unit nearly every time. GPT-5 mini caught them in 4 of 50 backlogs and 5 of 50 backlogs.", "The duplicate row told them apart. Claude Opus 5.5 caught it in 46 of 50 backlogs, Claude Fable 5.1 in 40, Claude Opus 5 in 29, Claude Sonnet 5 in 3, and Claude Haiku 4.5, GPT-5 mini and Gemini 3.1 Flash Lite in none.", "Loading a skill from a public catalog, measured against no help, went like this.", "Claude Opus 5: no measured difference with product manager toolkit and feature prioritization assistant; a loss too small to be sure of with define prioritization framework.", "Claude Sonnet 5: no measured difference with every skill.", "Claude Haiku 4.5: a gain too small to be sure of with every skill.", "GPT-5 mini: no measured difference with product manager toolkit and feature prioritization assistant; no score with define prioritization framework (it asked for a value before scoring on 45 of 50 answers).", "Gemini 3.1 Flash Lite: no measured difference with product manager toolkit and feature prioritization assistant; a loss too small to be sure of with define prioritization framework.", "Against the one plain sentence, no skill did better on any model. Define prioritization framework on Claude Opus 5 and define prioritization framework on Gemini 3.1 Flash Lite did worse.", "Claude Fable 5.1 and Claude Opus 5.5 got no skill runs: nothing left to measure on the trial round." ], "advice": [ "On Claude Opus 5, Claude Sonnet 5, Claude Fable 5.1, Claude Opus 5.5 and Gemini 3.1 Flash Lite, trust the arithmetic.", "Gemini 3.1 Flash Lite's misses sit in one backlog, where it got 26 of 37 rows right every time it was asked. Do not read its average as the same on every backlog.", "On Claude Haiku 4.5 and GPT-5 mini, check every score.", "No model caught the duplicate every time, so check for duplicates yourself whichever you use.", "Telling the model what kind of task it is helped Claude Haiku 4.5 (+4 rows per backlog, a gain too small to be sure of). It made no measured difference on Claude Opus 5, Claude Sonnet 5, Claude Fable 5.1, Claude Opus 5.5, GPT-5 mini and Gemini 3.1 Flash Lite.", "Do not load one of these skills for this job: none did better than the one plain sentence on any model, and two did worse." ], "useWhen": [ "RICE scoring of a backlog whose inputs are complete, on Claude Opus 5 (tested in Claude Code), Claude Sonnet 5 (tested in Claude Code), Claude Fable 5.1 (tested in Claude Code), Claude Opus 5.5 (tested in Claude Code) and Gemini 3.1 Flash Lite (tested via API): each got at least nine rows in ten right with no help.", "Spotting a duplicate item as well, on Claude Opus 5.5 (tested in Claude Code): it caught the planted duplicate in at least nine backlogs in ten." ], "skipWhen": [ "Unchecked RICE arithmetic on Claude Haiku 4.5 (tested in Claude Code) and GPT-5 mini (tested via API): Claude Haiku 4.5 got 7 of 22 rows right and GPT-5 mini got 7 of 12 rows right.", "Relying on Claude Haiku 4.5 (tested in Claude Code), GPT-5 mini (tested via API) and Gemini 3.1 Flash Lite (tested via API) to notice a duplicate: none caught it in any backlog.", "Relying on Claude Opus 5 (tested in Claude Code), Claude Sonnet 5 (tested in Claude Code) and Claude Fable 5.1 (tested in Claude Code) to notice a duplicate: Claude Opus 5 caught it in 29 of 50 backlogs, Claude Sonnet 5 caught it in 3 of 50 backlogs and Claude Fable 5.1 caught it in 40 of 50 backlogs.", "Treating Gemini 3.1 Flash Lite's average as uniform: one backlog sat at 26 of 37 rows right.", "Do not load one of these skills for this job: none did better than the one plain sentence on any model, and two did worse.", "Any job this page does not measure. This record covers RICE scoring and ranking of a supplied backlog with three planted data problems, and nothing else." ], "modelsByOutcome": { "trustTheArithmetic": [ "claude-opus-5", "claude-sonnet-5", "claude-fable-5-1", "claude-opus-5-5", "gemini-3.1-flash-lite" ], "checkEveryScore": [ "claude-haiku-4-5", "gpt-5-mini" ], "catchesTheDuplicate": [ "claude-opus-5-5" ], "sometimesCatchesTheDuplicate": [ "claude-opus-5", "claude-sonnet-5", "claude-fable-5-1" ], "missesTheDuplicate": [ "claude-haiku-4-5", "gpt-5-mini", "gemini-3.1-flash-lite" ], "unevenAcrossBacklogs": [ "gemini-3.1-flash-lite" ] }, "models": [ { "model": "claude-opus-5", "name": "Claude Opus 5", "instrument": "claude-code", "instrumentLabel": "tested in Claude Code", "backlog": "40-row backlog", "rowsRightWithNoHelp": "36 of 37 rows", "accuracy": 0.972972972972973, "duplicateCaught": "29 of 50 backlogs", "missingConfidenceCaught": "50 of 50 backlogs", "oddUnitCaught": "50 of 50 backlogs", "costPerBacklogUsd": 0.04838374, "costLabel": "CLI-reported estimate, not billed", "instructionEffect": "No measured effect", "skillRuns": "Skill runs done" }, { "model": "claude-sonnet-5", "name": "Claude Sonnet 5", "instrument": "claude-code", "instrumentLabel": "tested in Claude Code", "backlog": "40-row backlog", "rowsRightWithNoHelp": "36 of 37 rows", "accuracy": 0.9621621621621623, "duplicateCaught": "3 of 50 backlogs", "missingConfidenceCaught": "50 of 50 backlogs", "oddUnitCaught": "49 of 50 backlogs", "costPerBacklogUsd": 0.036682775999999986, "costLabel": "CLI-reported estimate, not billed", "instructionEffect": "No measured effect", "skillRuns": "Skill runs done" }, { "model": "claude-haiku-4-5", "name": "Claude Haiku 4.5", "instrument": "claude-code", "instrumentLabel": "tested in Claude Code", "backlog": "25-row backlog", "rowsRightWithNoHelp": "7 of 22 rows", "accuracy": 0.3209090909090909, "duplicateCaught": "0 of 50 backlogs", "missingConfidenceCaught": "50 of 50 backlogs", "oddUnitCaught": "50 of 50 backlogs", "costPerBacklogUsd": 0.004410900000000001, "costLabel": "CLI-reported estimate, not billed", "instructionEffect": "Seemed to help a little, but too small to be sure of", "skillRuns": "Skill runs done" }, { "model": "claude-fable-5-1", "name": "Claude Fable 5.1", "instrument": "claude-code", "instrumentLabel": "tested in Claude Code", "backlog": "40-row backlog", "rowsRightWithNoHelp": "37 of 37 rows", "accuracy": 1, "duplicateCaught": "40 of 50 backlogs", "missingConfidenceCaught": "50 of 50 backlogs", "oddUnitCaught": "50 of 50 backlogs", "costPerBacklogUsd": 0.15075642, "costLabel": "CLI-reported estimate, not billed", "instructionEffect": "Could not measure", "skillRuns": "Nothing left to measure" }, { "model": "claude-opus-5-5", "name": "Claude Opus 5.5", "instrument": "claude-code", "instrumentLabel": "tested in Claude Code", "backlog": "40-row backlog", "rowsRightWithNoHelp": "37 of 37 rows", "accuracy": 0.9994594594594595, "duplicateCaught": "46 of 50 backlogs", "missingConfidenceCaught": "50 of 50 backlogs", "oddUnitCaught": "50 of 50 backlogs", "costPerBacklogUsd": 0.056796548000000016, "costLabel": "CLI-reported estimate, not billed", "instructionEffect": "Could not measure", "skillRuns": "Nothing left to measure" }, { "model": "gpt-5-mini", "name": "GPT-5 mini", "instrument": "api", "instrumentLabel": "tested via API", "backlog": "15-row backlog", "rowsRightWithNoHelp": "7 of 12 rows", "accuracy": 0.5683333333333334, "duplicateCaught": "0 of 50 backlogs", "missingConfidenceCaught": "4 of 50 backlogs", "oddUnitCaught": "5 of 50 backlogs", "costPerBacklogUsd": 0.00060735, "costLabel": "billed at vendor list price", "instructionEffect": "No measured effect", "skillRuns": "Skill runs done" }, { "model": "gemini-3.1-flash-lite", "name": "Gemini 3.1 Flash Lite", "instrument": "api", "instrumentLabel": "tested via API", "backlog": "40-row backlog", "rowsRightWithNoHelp": "34 of 37 rows", "accuracy": 0.927027027027027, "duplicateCaught": "0 of 50 backlogs", "missingConfidenceCaught": "50 of 50 backlogs", "oddUnitCaught": "45 of 50 backlogs", "costPerBacklogUsd": 0.0019441999999999999, "costLabel": "billed at vendor list price", "instructionEffect": "No measured effect", "skillRuns": "Skill runs done" } ], "picks": [ { "instrument": "claude-code", "backlog": "40-row backlog", "best": [ "claude-fable-5-1" ], "tooCloseToCall": [ "claude-opus-5-5" ], "cheapestWithinRange": { "model": "claude-opus-5-5", "costPerBacklogUsd": null, "costFigure": "$0.05680", "costLabel": "CLI-reported estimate, not billed" }, "notRanked": [ { "model": "claude-haiku-4-5", "backlog": "25-row backlog" } ], "sentence": "Claude Fable 5.1 got the most rows right (37 of 37 rows); Claude Opus 5.5 is too close to call against it. Claude Opus 5.5 is the cheapest of those, at $0.05680 a backlog (CLI-reported estimate, not billed). Claude Haiku 4.5 was tested on an easier backlog and is not ranked against these." }, { "instrument": "api", "backlog": "", "best": [], "tooCloseToCall": [], "cheapestWithinRange": null, "notRanked": [ { "model": "gpt-5-mini", "backlog": "15-row backlog" }, { "model": "gemini-3.1-flash-lite", "backlog": "40-row backlog" } ], "sentence": "Each model was given backlogs at its own level of difficulty, so no one model is ranked first." } ], "excluded": [ { "model": "gpt-6-sol", "instrument": "codex-cli", "reason": "Codex is excluded from the 1a-i main runs and skill arms (amendment section 6)", "firstAttemptsWithTool": "33 of 60 first attempts", "scripts": 27, "askedBeforeAnswering": "1 of 25 with no help, 0 of 19 with the instruction" }, { "model": "gpt-6-luna", "instrument": "codex-cli", "reason": "Codex is excluded from the 1a-i main runs and skill arms (amendment section 6)", "firstAttemptsWithTool": "27 of 60 first attempts", "scripts": 20, "askedBeforeAnswering": "2 of 27 with no help, 1 of 21 with the instruction" } ], "skills": [ { "slug": "alirezarezvani-product-manager-toolkit", "name": "product manager toolkit", "pin": { "repo": "alirezarezvani/claude-skills", "path": "product-team/skills/product-manager-toolkit", "sha": "19392f7a08264ed00486a251f5b2098321771f94", "license": "MIT" }, "deferredToScript": "This skill ships scripts/rice_prioritizer.py and scripts/customer_interview_analyzer.py, which no arm receives, so its verdict is on the skill's text and references only.", "perModel": [ { "model": "claude-opus-5", "status": "Skill runs done", "result": { "rowsRight": "35 of 37 rows", "accuracy": 0.9551351351351351, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true, "stoppedToAskForScript": "3 of 50 backlogs", "atMainRunTableRule": { "rowsRight": null, "overNoHelp": null, "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true } } }, { "model": "claude-sonnet-5", "status": "Skill runs done", "result": { "rowsRight": "36 of 37 rows", "accuracy": 0.9702702702702704, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": false, "stoppedToAskForScript": "1 of 50 backlogs", "atMainRunTableRule": null } }, { "model": "claude-haiku-4-5", "status": "Skill runs done", "result": { "rowsRight": "11 of 22 rows", "accuracy": 0.49363636363636354, "withheld": null, "overNoHelp": "Seemed to help a little, but too small to be sure of", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } }, { "model": "claude-fable-5-1", "status": "Nothing left to measure", "result": null }, { "model": "claude-opus-5-5", "status": "Nothing left to measure", "result": null }, { "model": "gpt-5-mini", "status": "Skill runs done", "result": { "rowsRight": "7 of 12 rows", "accuracy": 0.6183333333333334, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } }, { "model": "gemini-3.1-flash-lite", "status": "Skill runs done", "result": { "rowsRight": "34 of 37 rows", "accuracy": 0.9108108108108107, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": false, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } } ] }, { "slug": "pmprompt-feature-prioritization-assistant", "name": "feature prioritization assistant", "pin": { "repo": "pmprompt/claude-plugin-product-management", "path": "skills/feature-prioritization-assistant", "sha": "3114dbe33fa2c38f781414c0b835023508db445f", "license": "MIT" }, "deferredToScript": null, "perModel": [ { "model": "claude-opus-5", "status": "Skill runs done", "result": { "rowsRight": "36 of 37 rows", "accuracy": 0.96, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true, "stoppedToAskForScript": "2 of 50 backlogs", "atMainRunTableRule": { "rowsRight": null, "overNoHelp": null, "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true } } }, { "model": "claude-sonnet-5", "status": "Skill runs done", "result": { "rowsRight": "36 of 37 rows", "accuracy": 0.9789189189189189, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": false, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } }, { "model": "claude-haiku-4-5", "status": "Skill runs done", "result": { "rowsRight": "10 of 22 rows", "accuracy": 0.4672727272727272, "withheld": null, "overNoHelp": "Seemed to help a little, but too small to be sure of", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } }, { "model": "claude-fable-5-1", "status": "Nothing left to measure", "result": null }, { "model": "claude-opus-5-5", "status": "Nothing left to measure", "result": null }, { "model": "gpt-5-mini", "status": "Skill runs done", "result": { "rowsRight": "8 of 12 rows", "accuracy": 0.6499999999999999, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } }, { "model": "gemini-3.1-flash-lite", "status": "Skill runs done", "result": { "rowsRight": "34 of 37 rows", "accuracy": 0.927027027027027, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": null, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } } ] }, { "slug": "product-on-purpose-define-prioritization-framework", "name": "define prioritization framework", "pin": { "repo": "product-on-purpose/pm-skills", "path": "skills/define-prioritization-framework", "sha": "1cef1a9eae10017389863d51e289e0ae41e17fcb", "license": "Apache-2.0" }, "deferredToScript": null, "perModel": [ { "model": "claude-opus-5", "status": "Skill runs done", "result": { "rowsRight": "32 of 37 rows", "accuracy": 0.8535135135135136, "withheld": null, "overNoHelp": "Seemed to do a little worse, but too small to be sure of", "overInstruction": "Seemed to do a little worse, but too small to be sure of", "oneSentenceGetsMostOfIt": false, "stoppedToAskForScript": "8 of 50 backlogs", "atMainRunTableRule": { "rowsRight": null, "overNoHelp": null, "overInstruction": "Seemed to do a little worse, but too small to be sure of", "oneSentenceGetsMostOfIt": false } } }, { "model": "claude-sonnet-5", "status": "Skill runs done", "result": { "rowsRight": "36 of 37 rows", "accuracy": 0.975135135135135, "withheld": null, "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": false, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": { "rowsRight": "35 of 37 rows", "overNoHelp": "No measured effect", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": false } } }, { "model": "claude-haiku-4-5", "status": "Skill runs done", "result": { "rowsRight": "11 of 22 rows", "accuracy": 0.5076515151515151, "withheld": null, "overNoHelp": "Seemed to help a little, but too small to be sure of", "overInstruction": "No measured effect", "oneSentenceGetsMostOfIt": true, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } }, { "model": "claude-fable-5-1", "status": "Nothing left to measure", "result": null }, { "model": "claude-opus-5-5", "status": "Nothing left to measure", "result": null }, { "model": "gpt-5-mini", "status": "Skill runs done", "result": { "rowsRight": null, "accuracy": null, "withheld": { "reason": "no answer that scored on item-01, item-02, item-04, item-07, item-08", "askedBeforeAnswering": 45, "of": 50, "notInRequestedFormat": 5, "sentence": "GPT-5 mini with define prioritization framework loaded asked for a value before scoring on 45 of 50 answers and gave 5 outside the requested format. So 5 of 10 backlogs never got a score and no rows-right figure is given." }, "overNoHelp": "the product-on-purpose-define-prioritization-framework arm's accuracy is withheld: no answer that scored on item-01, item-02, item-04, item-07, item-08", "overInstruction": "the product-on-purpose-define-prioritization-framework arm's accuracy is withheld: no answer that scored on item-01, item-02, item-04, item-07, item-08", "oneSentenceGetsMostOfIt": null, "stoppedToAskForScript": null, "atMainRunTableRule": null } }, { "model": "gemini-3.1-flash-lite", "status": "Skill runs done", "result": { "rowsRight": "33 of 37 rows", "accuracy": 0.8810810810810811, "withheld": null, "overNoHelp": "Seemed to do a little worse, but too small to be sure of", "overInstruction": "Seemed to do a little worse, but too small to be sure of", "oneSentenceGetsMostOfIt": false, "stoppedToAskForScript": "0 of 50 backlogs", "atMainRunTableRule": null } } ] } ], "tableRuleCorrection": { "sentence": "The rule for finding the scored table in an answer was corrected after the runs: a short recap table after the full one no longer counts as the answer. The corrected reading is the one every figure on this page uses. It changed the score of 3 of 1450 answers: Claude Opus 5 with one plain instruction, one backlog, from 5 of 37 rows right to 20 of 37 rows; Claude Sonnet 5 with the skill loaded (define prioritization framework), one backlog, from 2 of 37 rows right to 37 of 37 rows; Claude Sonnet 5 with the skill loaded (define prioritization framework), one backlog, from 3 of 37 rows right to 37 of 37 rows. Each figure those answers move is shown with the figure under the earlier reading beside it.", "rule": "The last-table coverage tie-break was adopted after the main runs, so it is a published correction, not a silent rescore. The corrected rule is primary. Any figure it moves is shown with the figure under the rule in force at main-run time beside it, the correction stated next to the figure, linking the amendment's list of every changed record (ordinal, model, arm, pin, old and new score).", "changedAnswers": 3, "ofAnswers": 1450, "changes": [ { "phase": "main", "ordinal": 13, "model": "claude-opus-5", "arm": "instruction", "pin": null, "item": "item-02", "oldScore": 0.13513513513513514, "newScore": 0.5405405405405406 }, { "phase": "skill", "ordinal": 230, "model": "claude-sonnet-5", "arm": "skill", "pin": "product-on-purpose-define-prioritization-framework", "item": "item-06", "oldScore": 0.05405405405405406, "newScore": 1 }, { "phase": "skill", "ordinal": 242, "model": "claude-sonnet-5", "arm": "skill", "pin": "product-on-purpose-define-prioritization-framework", "item": "item-07", "oldScore": 0.08108108108108109, "newScore": 1 } ] }, "disclosures": { "provenance": { "rationale": "Andy chose AI drafting for RICE wording because teams increasingly generate tickets with AI from team conversations; an AI-drafted backlog reflects the input PMs now hand to assistants.", "backlogTitles": "AI-drafted, human-reviewed. Andy wrote the seed lines (the example tickets) and the point scale; an AI model (Claude Opus) expanded them into the full title list, which Andy reviewed and adopted.", "duplicatePairs": "AI-drafted, human-reviewed. An AI model (Claude Opus) wrote the second wordings of titles expanded from Andy's seed lines; Andy reviewed and adopted them.", "ask": "AI-drafted, human-reviewed. An AI model (Claude Opus) drafted the ask to Andy's stated requirements; Andy reviewed and adopted it.", "registeredRisk": "Claude-family models read titles drafted in their own family's style. Gold is arithmetic on seeded values, so the expected effect is small.", "duplicatePairsRisk": "The duplicate rows' second wordings are also drafted in a Claude-family style, and spotting the duplicate is a scored trap, so for that trap the effect need not be small.", "askRisk": "The ask is AI-drafted in a Claude-family style and reaches every arm, so a Claude-style instruction is a family-level risk across all cells, not a risk to one trap.", "duplicateTrapResults": "Duplicate-trap results are published split by model family, with the Claude-style drafting note next to the Claude figures, not footnoted." }, "duplicateWordList": "A duplicate counts as caught when the answer names the row with a word from a fixed list. That list was widened by four phrasings (\"same work\", \"duplication\", \"deduplicate\" and \"identical to\") after the trial round's answers were read and before any main run. In the trial round the wider list moved: Claude Fable 5.1 with no help, 5 of 10 backlogs caught to 7 of 10 backlogs; Claude Opus 5.5 with no help, 7 of 10 backlogs caught to 9 of 10 backlogs; Claude Opus 5.5 with one plain instruction, 9 of 10 backlogs caught to 10 of 10 backlogs; Gemini 3.1 Flash Lite with one plain instruction, 0 of 10 backlogs caught to 1 of 10 backlogs. Every duplicate figure on this page is under the wider list.", "skillRunDecisions": [ { "model": "claude-opus-5", "decision": "Skill runs done", "disclosure": "The decision to run skills on Claude Opus 5 was taken on the trial round, under a rule widened after that round was read: a model gets skill runs if it was under 90% on rows right or under 90% on problems flagged. In the trial round Claude Opus 5 got 31 of 37 rows right and flagged 25 of 30 problems. In the main runs it got 36 of 37 rows right and flagged 129 of 150 problems; those figures decide nothing." }, { "model": "claude-sonnet-5", "decision": "Skill runs done", "disclosure": "The decision to run skills on Claude Sonnet 5 was taken on the trial round, under a rule widened after that round was read: a model gets skill runs if it was under 90% on rows right or under 90% on problems flagged. In the trial round Claude Sonnet 5 got 36 of 37 rows right and flagged 19 of 30 problems. In the main runs it got 36 of 37 rows right and flagged 102 of 150 problems; those figures decide nothing." }, { "model": "claude-haiku-4-5", "decision": "Skill runs done", "disclosure": "The decision to run skills on Claude Haiku 4.5 was taken on the trial round, under a rule widened after that round was read: a model gets skill runs if it was under 90% on rows right or under 90% on problems flagged. In the trial round Claude Haiku 4.5 got 5 of 22 rows right and flagged 20 of 30 problems. In the main runs it got 7 of 22 rows right and flagged 100 of 150 problems; those figures decide nothing." }, { "model": "claude-fable-5-1", "decision": "Nothing left to measure", "disclosure": "The decision to run skills on Claude Fable 5.1 was taken on the trial round, under a rule widened after that round was read: a model gets skill runs if it was under 90% on rows right or under 90% on problems flagged. In the trial round Claude Fable 5.1 got 37 of 37 rows right and flagged 27 of 30 problems under the wider wording list for the duplicate (25 of 30 problems under the list as first written). Under the list as first written it would have had room on problems flagged, and skill runs; the wider list put it at the 90% line, and it got none. In the main runs it got 37 of 37 rows right and flagged 140 of 150 problems; those figures decide nothing." }, { "model": "claude-opus-5-5", "decision": "Nothing left to measure", "disclosure": "The decision to run skills on Claude Opus 5.5 was taken on the trial round, under a rule widened after that round was read: a model gets skill runs if it was under 90% on rows right or under 90% on problems flagged. In the trial round Claude Opus 5.5 got 37 of 37 rows right and flagged 29 of 30 problems under the wider wording list for the duplicate (27 of 30 problems under the list as first written). In the main runs it got 37 of 37 rows right and flagged 146 of 150 problems; those figures decide nothing." }, { "model": "gpt-5-mini", "decision": "Skill runs done", "disclosure": "The decision to run skills on GPT-5 mini was taken on the trial round, under a rule widened after that round was read: a model gets skill runs if it was under 90% on rows right or under 90% on problems flagged. In the trial round GPT-5 mini got 7 of 12 rows right and flagged 5 of 30 problems. In the main runs it got 7 of 12 rows right and flagged 9 of 150 problems; those figures decide nothing." }, { "model": "gemini-3.1-flash-lite", "decision": "Skill runs done", "disclosure": "The decision to run skills on Gemini 3.1 Flash Lite was taken on the trial round, under a rule widened after that round was read: a model gets skill runs if it was under 90% on rows right or under 90% on problems flagged. In the trial round Gemini 3.1 Flash Lite got 34 of 37 rows right and flagged 19 of 30 problems. In the main runs it got 34 of 37 rows right and flagged 95 of 150 problems; those figures decide nothing." } ], "reviewFixes": { "postCalibrationRules": "The skill-arm cells were fixed from calibration run 2 under two rules amended after that run's answers were read and before any main run: the headroom rule (room on accuracy or on flag rate, in place of accuracy alone) and the duplicate lexicon (four entries added to the fixed lexicon of the job document). Each verdict that rests on them says so beside the verdict, not in a fold, with the calibration figure it was decided on and the main run's own none-arm figure next to it.", "duplicateFlagRatesBothLexicons": "Duplicate-trap flag rates are published under both lexicons, the job document's original and the amended one, as counts over trap outcomes, so a reader can see what the amendment moved.", "headroomTable": "The headroom table shows counts, not rates alone, and shows Fable 5.1 as borderline: none-arm flag rate 0.900, 27 of 30 outcomes, exactly at the threshold under the amended lexicon (25 of 30 under the original), beside Opus 5.5 at 0.967 (29 of 30). It states that the main runs independently support both exclusions: on the main data, Fable 5.1's none arm is at 1.000 accuracy and 0.933 flag rate, and Opus 5.5's at 0.999 and 0.973.", "deferredToScript": "An answer that stopped short and deferred the arithmetic to running a script or code, with only a sample or partial table, is published as its own descriptive outcome, beside the accuracy it scored, per model, arm and pin, on every arm. A low accuracy on such an answer is a model that stopped to ask for a tool, not wrong arithmetic.", "skillCardScriptNote": "A skill card whose skill expects scripts (the pin's displayNote says it ships scripts that no arm receives) carries a one-line note that the result measures the skill's text alone: the instrument runs with no tools, and the pin injects the skill's text and references only.", "tableRuleCorrection": "The last-table coverage tie-break was adopted after the main runs, so it is a published correction, not a silent rescore. The corrected rule is primary. Any figure it moves is shown with the figure under the rule in force at main-run time beside it, the correction stated next to the figure, linking the amendment's list of every changed record (ordinal, model, arm, pin, old and new score)." }, "limits": { "instruments": "Instruments are never pooled, averaged or subtracted; a Claude Code figure beside an API figure is two instruments (cohort section 9).", "ranking": "Lifts compare within a model across arms and pins, never across models as a ranking (R27; cohort 11.2).", "codeExecution": "No instrument has code execution, so no instrument difference is attributable to it (rice section 8).", "trapColumns": "Exact-score accuracy alone does not say a model got RICE right; the trap outcomes publish beside it on every row (rice sections 5 and 10).", "intervals": "The unit of every interval is the item; passes are repeat samples of one item (cohort section 2).", "headroom": "The skill-arm cells were fixed from calibration run 2 under the amended headroom rule; the main run's none-arm figures are reported beside that reading and decide nothing (amendment section 12).", "trapDetector": "Trap outcomes come from the registered pattern detector with the lexicon as amended in calibration data before any main run; its hand audit is amendment 1a-i-main-run-conditions sections 4, 11.2 and 11.3." } }, "evidence": { "record": "https://github.com/rampstack/openaddict-app/blob/main/harness/results/pm-wave-1/stage-1a-i-results.json", "recordPath": "harness/results/pm-wave-1/stage-1a-i-results.json", "answers": { "path": "harness/results/pm-wave-1/stage-1a-i-main/records.jsonl", "sha256": "42d3c803898908805a8df31d6c01d38607bfe687b50a9aeb4f522ac74f6f9fb1", "count": 700, "valid": 700 }, "skillAnswers": { "path": "harness/results/pm-wave-1/stage-1a-i-skill/records.jsonl", "sha256": "0a6e1de54a6562208003debf2f549e1b0b24e4e8d25ea9ff7a0de1c8950423fd", "count": 900, "valid": 750 }, "rescoreCheck": { "main": { "tableRule": "last", "records": 700, "identical": 700, "differing": [] }, "skill": { "tableRule": "last", "records": 750, "identical": 750, "differing": [] } }, "generatedBy": "harness/src/pm-wave-1/main-1a-analysis.ts", "registeredBy": [ "docs/preregistration/pm-wave-1-cohort.md sections 2, 5, 8, 9, 11, 12", "docs/preregistration/pm-wave-1-rice-prioritization.md sections 1, 3, 5, 8, 10", "docs/preregistration/pm-wave-1-amendment-D3-edge-cases-20261005.md sections 1, 2", "docs/preregistration/pm-wave-1-amendment-Q1-rice-titles-20261005.md section 8.5", "docs/preregistration/pm-wave-1-amendment-1a-i-ask-backlog-in-message-20261010.md section 7", "docs/preregistration/pm-wave-1-amendment-1a-i-main-run-conditions-20261011.md sections 3, 4, 7, 11.3, 12", "docs/preregistration/pm-wave-1-amendment-1a-i-review-fixes-20261012.md sections 1, 3, 4" ], "verdictRule": { "module": "lib/verdict/rule.ts", "rule": "status_v1 classify, called by the page on each lift's scale, baselineMean, treatmentMean, delta and halfWidth when it reads this record; no verdict word is stored here (cohort 11.2: called and not reproduced)", "scale": "unit", "interval": "1.96 * sd(per-item differences of per-item arm means) / sqrt(pairs), as lib/skills/cells.ts computes it" }, "scoring": "every valid record re-scored from its raw output by the scorer at this commit under the corrected table rule (amendment 1a-i-review-fixes section 4: the last table naming more than half of S, else the largest), which is primary, with asked before answering and the amended duplicate lexicon of amendment 1a-i-main-run-conditions; any figure that differs under the rule in force at main-run time (the plain last table, under which the stored scores were written) carries that figure beside it; rescoreCheck holds the stored scores against a re-score under that rule" } }