Methodology

How We Test Prompting Tips and Skills

This is how we test, in short. The full method is one link away.

What we test, and how we decide

  1. We test tips for writing prompts, and skills you can load into a model.
  2. Each test is a set of tasks, and we write down the right answers first.
  3. We run each task with the tip or skill, and again without it.
  4. A script or a second model scores each answer, and each page says which.
  5. We test each model on its own, by its full name and version.
  6. We fix the rule for what counts as a result before we run a single task.
  7. The rule looks at the gap between the two scores and how far chance could move it.
  8. If the gap is clear, we say the tip or skill helped, or made things worse.
  9. If the gap is too small to trust, we say the result is unclear.
  10. If a model already scores near the top with no help, we say there was nothing left to measure.

Five rules we never break

  • The same task, with and without. Every result compares the same tasks, run once with the tip or skill and once without it.
  • A named model and version. A result belongs to one model, named in full. We never report a result for AI in general.
  • Never averaged across test methods. We test through the API, Claude Code and Codex. Each gives the model different tools, so their results are kept apart.
  • Every task, or held back. If too many answers are missing, we hold the result back and say why, rather than report part of it.
  • Corrections in public. When we get something wrong, we fix it and list the fix where anyone can read it.

We have made 13 corrections so far. Read every one.

How to cite this

To cite one result, link to its page. Name the model and the test method, as the page does. To cite the method, use this line:

Open Addict, “How We Test Prompting Tips and Skills”, https://openaddict.com/methodology

Add the day you read it, since the method can change. Every change is listed.

What’s behind this

By Andy Dunn.