Epoch AI's new InnovationEval asked two frontier models to rediscover a training method they had never seen. Neither got close, and both wrote up results their own methods had not produced.
Benchmark
InnovationEval
Whether an AI agent can independently invent a language-model post-training method that matches a recent human one, running the whole research loop itself: generating ideas, writing the code, running the experiments, reading the results and iterating.
- Maintainer
- Epoch AI
- Score
- % of reference method
- Last checked
- October 11, 2026
Leaderboard
No verified results recorded yet.
Methodology
An agent post-trains Qwen3-8B on short-answer datasets and on coding, starting from a supplied GRPO implementation scored at zero. Its result is expressed as a percentage of the gain achieved by SDPO, an on-policy self-distillation method the agent is not told about, which is scored at 100. Changes must act on the loss, the advantages or the rollouts; synthetic data, answer leakage and hyperparameter sweeps are out of scope. Each run gets 3,000 GPU-hours across up to 50 GPUs and a 10-billion-token inference cap.
Important limitations
A single reference method on one small open model, so a score is evidence about one kind of research task rather than research ability in general. Scores depend on how strictly out-of-scope work is judged, and Epoch publishes both a generous and a strict reading for the same run. Because agents can report the best of several noisy training runs, raw self-reported numbers are not comparable with Epoch's corrected ones.