The same questions, twelve different askers
A paper posted on arXiv on 30 October put four smoking-cessation questions to six widely deployed large language models from four providers, asking each question as twelve different user personas. The personas varied by life stage, by self-described expertise and by smoking history. The run produced 2,880 responses.
The authors chose smoking cessation deliberately: it is a domain with codified clinical guidance and large public-health stakes, which makes it possible to say what a complete answer contains rather than only whether an answer sounds helpful.
The teen gap
The clearest result is about age. In every one of the six models, a persona presenting as a teenager was told about FDA-approved prescription cessation medications less often than an adult persona — by between 32 and 67 percentage points.
That is not a small drift in tone. It is a category of treatment that one group of users is systematically less likely to hear about at all.
The expertise gap runs the other way
The second finding is less intuitive. In five of the six models, a persona who presented healthcare expertise received fewer facts than a persona who presented none.
A model appears to read professional self-description as permission to skip the basics. That is defensible as conversational behaviour and risky as information policy, because the model has no way to confirm the claimed expertise — a persona is simply a line of text.
A prompt that narrows the gap, unevenly
The authors tested one deployable fix: a short system prompt specifying a “factual floor”, the set of core medical facts every user should receive when relevant, while leaving tone, vocabulary and framing free to adapt to the reader.
It worked in the narrow sense. The prompt reduced persona differences in every model tested. But its effect on overall fact coverage varied across models in both size and direction — in some systems coverage rose, in others it fell. The conclusion the authors draw is the operational one: a system prompt validated on one model cannot be assumed to transfer to another.
What the paper does not claim
It stops short of saying every group should hear identical content. The paper cautions that equal mention rates are not always the correct target, which matters: prescription cessation medication guidance genuinely differs by age, and a model that adjusted correctly would also show a gap. What the audit establishes is the size of the gap and that it is consistent across vendors, not that each individual answer was wrong.
What to watch
Six models from four providers all showing the same age-linked gap points at something in common — training data, safety tuning for minors, or both — rather than at one vendor’s choices. The useful next study would separate those, and would run the same design on domains where the correct age adjustment is smaller. Until then, the practical finding for anyone deploying a health assistant is that a persona a user types into a box changes what the model tells them, and that a prompt fix validated elsewhere may not hold on their stack.
