RICE Framework with AI: How Accurate Is the Math?

What the test does

Which model for this job

Each pick is made inside one test method, among models given the same size of backlog, from what they did with no help.

How each model did with no help

Rows right is the share of the backlog scored to the exact RICE figure, said as rows of the backlog less its three planted rows. Problems flagged counts the three planted problems over every answer. The same backlog was asked five times; the last column counts the backlogs that got the same score every time.

Exact-score accuracy alone does not say a model got RICE right; the trap outcomes publish beside it on every row (rice sections 5 and 10).

An answer that stopped short and deferred the arithmetic to running a script or code, with only a sample or partial table, is published as its own descriptive outcome, beside the accuracy it scored, per model, arm and pin, on every arm. A low accuracy on such an answer is a model that stopped to ask for a tool, not wrong arithmetic.

Catching bad data

Every backlog hides three problems, and nothing in the task says so. One item has no confidence figure, one item's effort is written in a different unit, and one item is listed twice under different wording. A model can flag the problem, guess past it as if the data were fine, or drop the row without a word.

The duplicate, by model family

Duplicate-trap results are published split by model family, with the Claude-style drafting note next to the Claude figures, not footnoted.

A duplicate counts as caught when the answer names the row with a word from a fixed list. That list was widened by four phrasings ("same work", "duplication", "deduplicate" and "identical to") after the trial round's answers were read and before any main run. In the trial round the wider list moved: Claude Fable 5.1 with no help, 5 of 10 backlogs caught to 7 of 10 backlogs; Claude Opus 5.5 with no help, 7 of 10 backlogs caught to 9 of 10 backlogs; Claude Opus 5.5 with one plain instruction, 9 of 10 backlogs caught to 10 of 10 backlogs; Gemini 3.1 Flash Lite with one plain instruction, 0 of 10 backlogs caught to 1 of 10 backlogs. Every duplicate figure on this page is under the wider list.

Duplicate-trap flag rates are published under both lexicons, the job document's original and the amended one, as counts over trap outcomes, so a reader can see what the amendment moved.

One plain sentence about the task

Each model was also asked with one sentence added, saying this is a backlog prioritization task to be done to a professional standard. The difference is in rows right per backlog, with the range it could take on another run.

• Claude Opus 5: no measured difference with the sentence (no change, which could be -2 rows to +1 rows on another run).
• Claude Sonnet 5: no measured difference with the sentence (no change, which could be -1 rows to +1 rows on another run).
• Claude Haiku 4.5 got +4 rows per backlog right with the sentence. It seemed to help a little, but the gain is too small to be sure of.
• Claude Fable 5.1 was at the top with and without the sentence, so there was nothing for the sentence to change.
• Claude Opus 5.5 was at the top with and without the sentence, so there was nothing for the sentence to change.
• GPT-5 mini: no measured difference with the sentence (+1 rows, which could be no change to +1 rows on another run).
• Gemini 3.1 Flash Lite: no measured difference with the sentence (no change, which could be -2 rows to +2 rows on another run).

Skills from public catalogs

Three skills written for this job were pinned from public catalogs. Each was loaded into the model and the same backlogs asked again, so the skill is measured against no help and against the one plain sentence. Which models got skill runs was decided on a trial round, and the decision and its figures are beside each model below.

The skill-arm cells were fixed from calibration run 2 under the amended headroom rule; the main run's none-arm figures are reported beside that reading and decide nothing (amendment section 12).

The skill-arm cells were fixed from calibration run 2 under two rules amended after that run's answers were read and before any main run: the headroom rule (room on accuracy or on flag rate, in place of accuracy alone) and the duplicate lexicon (four entries added to the fixed lexicon of the job document). Each verdict that rests on them says so beside the verdict, not in a fold, with the calibration figure it was decided on and the main run's own none-arm figure next to it.

The headroom table shows counts, not rates alone, and shows Fable 5.1 as borderline: none-arm flag rate 0.900, 27 of 30 outcomes, exactly at the threshold under the amended lexicon (25 of 30 under the original), beside Opus 5.5 at 0.967 (29 of 30). It states that the main runs independently support both exclusions: on the main data, Fable 5.1's none arm is at 1.000 accuracy and 0.933 flag rate, and Opus 5.5's at 0.999 and 0.973.

Each card below is one skill. Rows right is with the skill loaded; the two verdicts beside it compare that with no help and with the one plain sentence. The last column asks whether the one plain sentence got at least half of whatever the skill got.

• product manager toolkit

  Pinned from alirezarezvani/claude-skills at product-team/skills/product-manager-toolkit, MIT.

  This skill ships scripts/rice_prioritizer.py and scripts/customer_interview_analyzer.py, which no arm receives, so its verdict is on the skill's text and references only. A skill card whose skill expects scripts (the pin's displayNote says it ships scripts that no arm receives) carries a one-line note that the result measures the skill's text alone: the instrument runs with no tools, and the pin injects the skill's text and references only.

  Model	Result	Stopped to ask for a script	One sentence gets most of it?
  Claude Opus 5	35 of 37 rows against no help: No measured effect against the sentence: No measured effect	3 of 50 backlogs	Yes
  Claude Sonnet 5	36 of 37 rows against no help: No measured effect against the sentence: No measured effect	1 of 50 backlogs	No
  Claude Haiku 4.5	11 of 22 rows against no help: Seemed to help a little, but too small to be sure of against the sentence: No measured effect	0 of 50 backlogs	Yes
  Claude Fable 5.1	Nothing left to measure
  Claude Opus 5.5	Nothing left to measure
  GPT-5 mini	7 of 12 rows against no help: No measured effect against the sentence: No measured effect	0 of 50 backlogs	Yes
  Gemini 3.1 Flash Lite	34 of 37 rows against no help: No measured effect against the sentence: No measured effect	0 of 50 backlogs	No

  • Claude Opus 5: no measured difference with the skill loaded (-1 rows, which could be -2 rows to +1 rows on another run).
  • Claude Sonnet 5: no measured difference with the skill loaded (no change, which could be -1 rows to +1 rows on another run).
  • Claude Haiku 4.5 got +4 rows per backlog right with the skill loaded. It seemed to help a little, but the gain is too small to be sure of.
  • GPT-5 mini: no measured difference with the skill loaded (+1 rows, which could be -1 rows to +2 rows on another run).
  • Gemini 3.1 Flash Lite: no measured difference with the skill loaded (-1 rows, which could be -3 rows to +2 rows on another run).

• feature prioritization assistant

  Pinned from pmprompt/claude-plugin-product-management at skills/feature-prioritization-assistant, MIT.

  Model	Result	Stopped to ask for a script	One sentence gets most of it?
  Claude Opus 5	36 of 37 rows against no help: No measured effect against the sentence: No measured effect	2 of 50 backlogs	Yes
  Claude Sonnet 5	36 of 37 rows against no help: No measured effect against the sentence: No measured effect	0 of 50 backlogs	No
  Claude Haiku 4.5	10 of 22 rows against no help: Seemed to help a little, but too small to be sure of against the sentence: No measured effect	0 of 50 backlogs	Yes
  Claude Fable 5.1	Nothing left to measure
  Claude Opus 5.5	Nothing left to measure
  GPT-5 mini	8 of 12 rows against no help: No measured effect against the sentence: No measured effect	0 of 50 backlogs	Yes
  Gemini 3.1 Flash Lite	34 of 37 rows against no help: No measured effect against the sentence: No measured effect	0 of 50 backlogs	Not defined the skill lift is zero, so the ratio is undefined

  • Claude Opus 5: no measured difference with the skill loaded (no change, which could be -2 rows to +1 rows on another run).
  • Claude Sonnet 5: no measured difference with the skill loaded (+1 rows, which could be -1 rows to +2 rows on another run).
  • Claude Haiku 4.5 got +3 rows per backlog right with the skill loaded. It seemed to help a little, but the gain is too small to be sure of.
  • GPT-5 mini: no measured difference with the skill loaded (+1 rows, which could be no change to +2 rows on another run).
  • Gemini 3.1 Flash Lite: no measured difference with the skill loaded (no change, which could be -2 rows to +2 rows on another run).

• define prioritization framework

  Pinned from product-on-purpose/pm-skills at skills/define-prioritization-framework, Apache-2.0.

  Model	Result	Stopped to ask for a script	One sentence gets most of it?
  Claude Opus 5	32 of 37 rows against no help: Seemed to do a little worse, but too small to be sure of against the sentence: Seemed to do a little worse, but too small to be sure of	8 of 50 backlogs	No
  Claude Sonnet 5	36 of 37 rows 35 of 37 rows under the earlier reading against no help: No measured effect against the sentence: No measured effect	0 of 50 backlogs	No
  Claude Haiku 4.5	11 of 22 rows against no help: Seemed to help a little, but too small to be sure of against the sentence: No measured effect	0 of 50 backlogs	Yes
  Claude Fable 5.1	Nothing left to measure
  Claude Opus 5.5	Nothing left to measure
  GPT-5 mini	GPT-5 mini with define prioritization framework loaded asked for a value before scoring on 45 of 50 answers and gave 5 outside the requested format. So 5 of 10 backlogs never got a score and no rows-right figure is given.	not read	Not defined an arm is withheld
  Gemini 3.1 Flash Lite	33 of 37 rows against no help: Seemed to do a little worse, but too small to be sure of against the sentence: Seemed to do a little worse, but too small to be sure of	0 of 50 backlogs	No

  • Claude Opus 5 got -4 rows per backlog right with the skill loaded. It seemed to do a little worse, but the loss is too small to be sure of.
  • Claude Sonnet 5: no measured difference with the skill loaded (no change, which could be no change to +1 rows on another run).
  • Claude Haiku 4.5 got +4 rows per backlog right with the skill loaded. It seemed to help a little, but the gain is too small to be sure of.
  • GPT-5 mini with define prioritization framework loaded asked for a value before scoring on 45 of 50 answers and gave 5 outside the requested format. So 5 of 10 backlogs never got a score and no rows-right figure is given.
  • Gemini 3.1 Flash Lite got -2 rows per backlog right with the skill loaded. It seemed to do a little worse, but the loss is too small to be sure of.