TheoPhysEval was built from problems working physicists had not published, precisely so no model could have seen them. The best agent missed one, at an average cost of about $2.33 a problem.
Benchmark
TheoPhysEval
How many of 50 unpublished, research-level theoretical physics problems with verifiable answers an off-the-shelf AI agent solves on its own, without hints from a human.
- Maintainer
- A. Ramesh Chandra and 36 co-authors
- Score
- problems solved out of 50
- Last checked
- October 15, 2026
Leaderboard
No verified results recorded yet.
Methodology
Each problem was written by a working physicist, drawn from unpublished work, and has a verifiable answer. Agents run in two tracks: a realistic track with literature search and the ability to write and run scripts, and an isolated track with no internet access. Each agent is given one run per track at maximum reasoning effort, and the authors record token usage and API cost alongside the pass rate.
Important limitations
A single run per agent means the figures carry real statistical uncertainty, and the authors found errors in several of their own reference solutions that initially counted against a correct model answer. The problems are mostly calculational rather than conceptual, so a high score shows competence at turning the crank, not research taste.