Giving the model a job title / myth-bust
Does role prompting work?
You have probably heard you should tell the model to act as an expert. We tested it. Here is what we found.
Giving the model a job title did not change answer quality in our tests. Write a clearer question instead.
Why it matters
Role lines take up room in your prompt and give you nothing back.
What to do instead
Spend those words on the task. Say what you want, who will read it, and what a good answer has to cover.
Debunked
Rubric mean moved at most 0.13 points on all three models, inside the margin every time.
tested via API · No measured effect on Claude Haiku 4.5, tested via API · No measured effect on GPT-5 mini, tested via API · No measured effect on Gemini 3.1 Flash Lite, tested via API
What the marks mean
- No measured effect
See the numbers per model
Without to with, per model
Via API tested via API
GPT-5 mini
No measured effect
Gemini 3.1 Flash Lite
No measured effect
Claude Haiku 4.5
No measured effect
rubric mean, 1 to 5
Show per-task detail
Every item, without and with, per model
Via API tested via API
- Claude Haiku 4.5
- Gemini 3.1 Flash Lite
- GPT-5 mini
rubric mean, 1 to 5
Hover or focus an item to read its task and every model’s two values.
Pooled: the interval is taken over the records, not these means. These are those same records collapsed per item, so where an item carries more passes than another the lines need not average to the figure the table prints.
The item means behind this plot
| Item | Claude Haiku 4.5 | Gemini 3.1 Flash Lite | GPT-5 mini | |||
|---|---|---|---|---|---|---|
| Task | without | with | without | with | without | with |
| t03 | 3.200 | 2.917 | 4.133 | 4.333 | 3.800 | 3.933 |
| t04 | 3.133 | 3.133 | 4.000 | 4.000 | 4.133 | 4.200 |
| t05 | 3.600 | 3.867 | 3.667 | 4.333 | 4.400 | 4.800 |
| t01 | 3.800 | 4.000 | 4.067 | 4.067 | 4.000 | 4.067 |
| t02 | 3.667 | 3.667 | 4.133 | 4.000 | 4.067 | 4.133 |
| t06 | 4.267 | 3.867 | 4.933 | 4.000 | 4.667 | 4.733 |
Result
Role assignment moved the rubric mean by at most 0.13 points on all three models tested via API, inside the margin every time. Measured 2026-08-12.
Show the per-model numbers
Claim tested: Role assignment improves response quality on domain questions.
No measured effect on Claude Haiku 4.5, GPT-5 mini and Gemini 3.1 Flash Lite.
Circulates in practitioner communities. Tested because it circulates, not because it is endorsed.
This tip is OpenAddict's plain-language read of the measured result. The measurement below is the evidence, and it is what the reading has to answer to.
Ledger idC09-role-assignment
Clarification, . Until 2026-08-31 this figure read 5, which was the pass count rather than a number of calls. What changed, and why it is filed once.
What was tested
This claim circulates in practitioner communities as advice about how to write prompts. That it circulates is an input to what gets tested here. It is a reason to test the claim, and it is not evidence for or against it. The result below is the evidence, and it is the only thing on this page that carries weight.
The comparison is paired. Two prompts differ in one respect, the manipulated variable, and are otherwise identical by construction. Nothing here supports a causal reading beyond that pairing.
Per-model numbers
| Measure | Claude Haiku 4.5 | GPT-5 mini | Gemini 3.1 Flash Lite |
|---|---|---|---|
| Control arm | 3.611n 30 | 4.178n 30 | 4.156n 30 |
| Treatment arm | 3.598n 29 | 4.311n 30 | 4.122n 30 |
| Delta | -0.013 | +0.133 | -0.033 |
| Interval, 95 percent | -0.265 to 0.238 | -0.046 to 0.312 | -0.200 to 0.133 |
| Orbit | In free drift59 of 60 records | In free drift60 of 60 records | In free drift60 of 60 records |
Every cell in this table is rubric graded. The score is a grader model reading the answer against a published rubric, not a deterministic check.
Orbit is assigned by the frozen status_v1 rule. On this scale, rubric mean, 1 to 5, the pass threshold is +0.30 and the failure floor is -0.30, each requiring an interval that excludes zero.
Coverage shortfall on this claim
Claude Haiku 4.5: 59 valid records of 60 attempted. 1 record carries a valid answer whose grader verdict could not be parsed, so it is in neither the valid count nor the invalid count and these columns do not sum to the arm total.
Method for this claim
- Task set
- 6 domain questions spread across law, medicine, tax, engineering, statistics, and security. Both arms ask the identical question; only the role sentence differs.
- Runs per model per arm
- 5 passes x 6 items
- Scoring
- Rubric graded, via rubric-quality-v1. A grader model reads each answer against a published rubric.
- Pass criterion as written for the pilot
- Mean graded score for treatment exceeds control by at least 0.5 rubric points on a 5 point scale, with the interval excluding zero.
The published verdict comes from status_v1, not from the pass criterion above. The criterion is recorded because it is what the claim was registered with before the run.
Model versions, as recorded
| Model | Version string returned | Delta on this claim | Interval |
|---|---|---|---|
| Claude Haiku 4.5 | claude-haiku-4-5-20251001 | -0.013 | -0.265 to 0.238 |
| GPT-5 mini | gpt-5-mini-2025-08-07 | +0.133 | -0.046 to 0.312 |
| Gemini 3.1 Flash Lite | gemini-3.1-flash-lite | -0.033 | -0.200 to 0.133 |
Reading across models
Sampling was not held constant across vendors, so comparing one model column against another compares two settings as well as two models.
GPT-5 mini rejected the fixed sampling setting and ran at its own default on all 1,040 of its calls. The other models ran at temperature 0.