What was published
The Allen Institute for AI has released FairStress, a benchmark for demographic fairness in model decisions, on 20 October. It is built from 300 two-candidate scenarios across 29 decision domains including hiring, lending, healthcare, education, housing and criminal justice, expanded into more than 13 million test cases. A 50,000-item subset, FairStress-Core, is selected using item response theory for faster runs. The dataset is on Hugging Face, the code is on GitHub, and Ai2 says it is being added to Olmo-eval.
The design has two axes. Demographic identity appears at four levels — absent, implied by a coded name, implied by context, or stated outright. And every scenario is paired with what the authors call adversarial prefixes: persuasive statements pushing for one candidate, each with a matched version pushing for the other. The target Ai2 is measuring is “symmetric fairness”: the same criteria, and the same resistance to being talked round, whichever group benefits.
The measurement finding
The most consequential result is methodological. Running the older BBQ benchmark, Ai2 reports that models generally leaned stereotype-consistent — but that changing the scoring setup, from fixed-option probabilities to generated answers, shifted several models toward minority-favouring behaviour. The direction of the measured bias, in other words, depended on how the benchmark was run.
That matters well beyond this paper. Fairness scores are cited in model cards, procurement decisions and regulatory filings, and a score whose sign moves with the harness is a weaker object than it looks.
What the models did
On FairStress itself, Ai2 reports that behaviour was close to symmetric when no demographic information was present, and that asymmetry grew as identity became more explicit. The pattern it found runs against the usual expectation: models were more willing to abandon a correct choice when the switch favoured a minority-coded candidate, a behaviour the authors call anti-stereotypical over-correction.
The gaps are specific. Ai2 puts the average gap in abandoning correct answers at 6.9 points with implied identity and 9.3 points with explicit labels. Where neither candidate was objectively stronger, it says frontier models chose the minority-coded candidate about 60 to 65% of the time. Under matched adversarial prefixes, switching toward a minority-coded candidate was 8.8 to 42.4 points more common than switching the other way, across twelve model-and-mode runs.
The explanations differed too: justifications for minority-coded choices used words such as “resilience” and “barriers” more often, while majority-coded choices drew more procedural language about criteria.
Fixes that did not work
Ai2 tested nine debiasing methods. Most did not reliably reduce the asymmetry, and some cost a great deal of accuracy: one activation-steering method dropped Qwen2.5-7B-Instruct from 98% to 47% on cases the authors describe as clear-cut. Counterfactual chain-of-thought narrowed the gap while largely preserving accuracy, but did not close it.
Caveats the authors state
FairStress encodes an equality-centred definition of fairness and does not capture equity-based views, which is a choice rather than a finding. The interpretability work — internal activations predicted which side a model would pick about 73% of the time, against 57 to 59% from text descriptions of those activations — covers only two models and a handful of demographic contrasts. An audit of Olmo 3’s training data found only nine of 28,000 topic clusters discussing merit-based decisions, all of them refusals, but Ai2 says whether those data patterns cause the behaviour is unresolved. Some reinforcement-learning framing estimates rest on as few as eight prompts.
What to watch
The useful next step is other people running it. The dataset and code are public and the Core subset is small enough to run cheaply, so the asymmetry figures are checkable in a way the BBQ result makes newly urgent: if a scoring choice can flip the sign of a fairness measurement, every published fairness number needs its harness attached.
