Reproduce it before you discover it
A paper posted to arXiv on 29 October argues that AI systems should demonstrate they can reproduce established scientific results before anyone trusts them to produce new ones. The benchmark it introduces, PAPERBENCHX, sets 93 expert-constructed paper-reproduction tasks across 12 research areas, including electromagnetics, photonics, chemistry, materials science, biology and robotics.
The headline result is narrow. Across ten frontier agent configurations, the best fully reproduced only 14% of tasks at the benchmark’s 90% threshold. Mean rubric scores across those configurations ran from 27.53% to 59.08% — enough to show real partial competence, and far short of a reliable result.
Why scoring the output is not enough
The design choice that distinguishes PAPERBENCHX from earlier reproduction benchmarks is what it grades. Submissions are replayed from scratch, and only the regenerated evidence is scored against expert-validated criteria. An agent cannot pass by describing a method convincingly or by handing in code that looks right; the code has to run and the numbers have to come back.
That matters because the failure mode the authors are testing for is exactly the one a text score hides. The paper frames the gap as the distance between understanding a paper and reliably producing reproducible evidence from it, and it is that second thing that scientific work depends on.
Extending beyond machine learning is the hard part
Existing reproduction benchmarks sit mostly inside AI and machine learning, where the tooling is familiar and the environments are uniform. The authors argue that moving outside that circle is harder for reasons that have little to do with reasoning ability: agents have to drive specialised scientific software, make physical and numerical modelling decisions, diagnose computations that run for a long time before failing, and separate valid evidence from output that merely looks plausible.
Each of those is a place where an agent that writes good code can still produce a wrong number without noticing. A simulation that converges to the wrong answer looks, on the surface, exactly like one that converges to the right one.
What the spread tells you
The distance between a 14% full-reproduction rate and mean rubric scores in the 27.53% to 59.08% band is the interesting part. Agents are getting substantial parts of these reproductions right — building environments, running experiments, producing output — while rarely getting all the way to a result that matches the paper. Partial credit is not what reproduction is for.
What to watch
The benchmark’s value will depend on whether its 93 tasks hold up as models improve, and on whether other groups can score against the same expert-validated criteria without the authors in the loop. The paper’s own conclusion is the one to carry forward: a large gap remains between understanding a paper and reliably producing reproducible evidence, and it is wider outside machine learning than inside it.
