One prompt change, a 52-point swing
The Allen Institute for AI released a benchmark on 27 October for a job that has quietly become load-bearing: having a language model decide whether a research idea is new. Ai2 calls it Old Ideas, Novel Problems, and its headline finding is that the judges are not stable.
On one prompt change, GPT-5.4’s accuracy on human-versus-generated pairs fell from 93.5% to 40.9%. The same kind of change moved Claude Opus 4.6 by 21 points on the human-only set. Across the generator-comparison tests, roughly a quarter to a third of verdicts changed on average. These are not borderline cases shifting at the margin; a judge that scores 40.9% on a two-way choice is doing worse than a coin.
How the test is built
Ai2 drew its human ideas from ICLR 2026 peer reviews, splitting them into a high-novelty pool of 154 ideas and a lower-novelty pool of 145. A second setting, Human+Generated, pairs the 154 human ideas against 154 ideas from a simple LLM generator. Judges are tested in two formats: pointwise, rating an idea on its own, and pairwise, choosing the more novel of two.
Six judges were run: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Ai2 published the dataset, the code and a pipeline for building further evaluation sets alongside the results.
The cheap baseline wins
The cost comparison is the part a research group will act on. A simple prompted Claude Opus 4.6 baseline scored 0.854 macro-F1 at $2.59. Idea Novelty Checker, a purpose-built evaluator, scored 0.642 at $30.66. Sakana’s AI Scientist evaluator scored 0.772 at $82.05. The plain prompt beat both tools on accuracy and cost at once.
Adding agency did not rescue it either. An agentic GPT-5.6 Sol judge reached 0.78 macro-F1, below the 0.89 that Claude Opus 4.6 managed. Ai2’s reading is that more search and more reasoning do not reliably improve novelty judgements.
Why this matters now
Automated novelty checks sit inside the idea-generation pipelines that labs and startups use to filter machine-generated research proposals, and increasingly inside reviewing workflows. If a judge’s verdict flips on a prompt rewrite, the filter is not measuring novelty — it is measuring the prompt.
Ai2 reports the brittleness is worse for AI-generated ideas than for human ones, which is precisely the case these systems are deployed to handle. The practical advice falling out of the paper is unflattering to the tooling market: before buying a novelty evaluator, test a plain prompt against it, and check whether its verdicts survive a rewrite of the instructions.
