Marking the forecasters’ homework
The Forecasting Research Institute has gone back over predictions it collected in 2022 and checked them against what happened. On artificial intelligence benchmarks, the panels were not close.
“Overall, superforecasters assigned an average probability of just 9.7% to the observed outcomes across these four AI benchmarks, compared to 24.6% from domain experts,” the institute writes in a review published on 23 September by Josh Rosenberg, Matt Reynolds, Nadja Flechner and Zack Devlin-Foltz.
The four benchmarks are MATH, MMLU, QuALITY and a gold medal at the International Mathematical Olympiad. The forecasts came from the Existential Risk Persuasion Tournament, run between June and October 2022, which put domain specialists and superforecasters — people with a track record of accurate prediction — on the same questions.
Where the gap was widest
The Olympiad result is the clearest case. AI reached gold-medal level in July 2025. The median expert had put that in 2030 and the median superforecaster in 2035 — five and ten years late respectively.
A separate FRI study on biosecurity produced the same shape. Asked when AI would match top virologist teams on a troubleshooting benchmark, the median expert said 2030 and the median superforecaster 2034. The institute writes that models likely matched that baseline as of April 2025 — within months of the survey being taken.
Why superforecasters did worse
The counterintuitive part is that the generalists with the better forecasting record were further off than the specialists. Superforecasting skill is built on base rates and on discounting hype, which is usually the right instinct and was the wrong one here. Domain experts were not right either; they were merely less wrong.
The review is also narrow in an important way. It scores benchmark and capability questions, where the record is poor, and it reports a more mixed picture on adoption. On the broad economic effects, it says, there is not yet enough resolved evidence to grade anyone.
What to do with it
The practical use of a review like this is calibration, not embarrassment. It suggests that when a panel of experts puts a capability a decade away, the distribution of outcomes has recently been skewed towards “sooner”, and that forecasts about AI capability should be read with that bias in mind.
It is one institute grading its own tournaments, and four benchmarks is a small sample. The test worth watching is the next round: FRI runs a rolling expert panel, and the same exercise repeated on forecasts made in 2026 will show whether the people answering have updated on being wrong.
