What is in it
OpenAI released MentalHealthBench on 23 September. It is an open set of 1,215 synthetic conversations about mental health, paired with 5,262 rubric criteria, written with more than 80 licensed psychologists and psychiatrists from 22 countries working across 19 languages.
The conversations are deliberately not all crises. They split 53.5 per cent non-acute — everyday worry, sleep, stress — 18.2 per cent high-acuity, and 28.3 per cent emergent, meaning someone’s immediate safety is at stake. The speakers are adults in 68.1 per cent of cases, teenagers in 21.2 per cent, clinicians in 5.8 per cent and caregivers in 4.9 per cent. Alongside English there are 105 Spanish conversations, 54 Hindi, 34 Arabic, 29 Portuguese, and smaller sets in German, Italian, Persian, Indonesian, Turkish and Chinese.
Criteria carry weights from -10 to +10, so a response can lose points for the wrong move rather than only gaining them for the right one. They are grouped under behaviours including seeking context, calibrating urgency, reality testing and leaving the user their own agency.
The scores, and who produced them
OpenAI’s launch table puts GPT-6 Astra at 57.3, GPT-6 Sol at 53.9, Claude Opus 5.5 at 52.4 and GPT-6 Luna at 50.2. Older models fall away sharply: GPT-4o from March 2025 at 32.1, Gemini 2.5 Pro at 29.5.
Every one of those figures is vendor-reported. OpenAI built the benchmark, wrote the rubric with its own recruited panel, ran the evaluation, and used one of its own models — GPT-5.6 Sol at high reasoning effort — as the grader. The two competing models in the table were scored by OpenAI, not by their makers.
That is not an accusation of bad faith. It is the standard caveat that applies to any vendor benchmark, and it applies here with extra force because the rubric encodes contestable clinical judgement rather than a right answer.
Why the open release matters more than the numbers
The useful part is that OpenAI has published the set rather than the leaderboard alone. Other researchers can run it, disagree with the rubric, regrade with a different model, or build on it. Examples carry a canary identifier, and OpenAI asks that they not be posted in plain text so that the set does not leak into training data.
That is the right shape for this kind of evaluation, and it is a genuine advance on crisis-response testing, which asks only whether a model says the right thing when someone states an intention to harm themselves. Most of the harm surface is not there. It is in the 53.5 per cent of ordinary conversations where a model can be subtly wrong for weeks.
What the benchmark cannot do is tell you whether anyone was helped. The conversations are synthetic. A high score means a response matched what a panel of clinicians said it should contain, which is a proxy for care and not a measure of it. The number worth waiting for is the first independent regrade.
