Fifty problems nobody had published
A group of physicists has built a benchmark out of 50 research-level theoretical physics problems that had never been published, each with a verifiable answer, and run five commercial AI agents at them with no human hints. The paper, TheoPhysEval, was posted to arXiv on 9 October. Its authors say a world expert in the relevant specialism could typically solve one of these problems in about a day or less, while an expert from a neighbouring field would take considerably longer.
The design matters more than the scores. The problems were drawn from the authors’ own unpublished work, which is the only reliable way to be sure a model has not simply read the answer. They span classical physics, stochastic processes and biological physics, non-equilibrium statistical mechanics, lattice theories, quantum phases of matter, electronic structure theory and quantum field theory.
The results
Each agent was run on two tracks: a realistic one with literature search and the ability to write and run scripts, and an isolated one with no internet access. On the realistic track, GPT-6-Astra running in Codex solved 49 of 50. Claude Opus 5.5 in the Claude Code CLI solved 46. Gemini 3.8 Flash in Antigravity solved 34, and the two open-weight entries, DeepSeek-V4.1-Flash and GLM 5.3, solved 27 and 21.
The costs are as striking as the scores. The authors infer $116.28 for Astra’s full run on the realistic track, which works out at about $2.33 a problem, against $240.43 for Opus and $37.42 for DeepSeek. GLM’s run was covered by a coding plan at an actual cost of $180 for both tracks; at API rates it would have been $311.71 and $371.46.
Taking the literature away barely helped
The finding the authors dwell on is the isolated track. Astra and Opus both solved 46 of 50 without any internet access — three fewer for Astra, exactly the same for Opus. Gemini scored 31, GLM 26 and DeepSeek 25. Many of these problems need specific formulas and notation from particular papers, and the authors write that a human could not attempt some of them without looking the background paper up. That the models can suggests, in the paper’s words, significant memorisation of the physics research literature.
The one problem Astra missed on the realistic track was an inverse problem — recovering a differential equation from a given curve — and no model solved it. Everything every other agent got right was a subset of what Astra got right.
What the authors themselves flag
Two caveats come from inside the paper. Each agent was run once per track, so the gap between the realistic and isolated tracks sits within the statistical uncertainty of the evaluation. And the authors initially scored Astra wrong on several problems before discovering that their own reference solutions contained subtle errors; once the experts corrected them, Astra’s answers were right every time.
The pass rates here are higher than other theoretical physics benchmarks report, which the authors note and say deserves explanation — they judge their own problems to be more advanced on inspection, and point to recent work suggesting some physics benchmarks are graded incorrectly. The dataset is published, so the discrepancy is checkable rather than a matter of trust.
