Measuring whether a model can discover something is hard because almost every benchmark has an answer the model may already have read. ExplorationBench, posted to arXiv on 25 September by researchers at Tencent Hunyuan with Fudan University and Tsinghua University, gets around that by building worlds whose rules deliberately contradict familiar knowledge.
Two sandboxes make up the benchmark. AlienCode is a small calculation language whose operators look familiar and behave differently, with 31 discovery targets and 70 tasks, graded by an interpreter on private inputs. AlienLogic contains 24 inference rules, and asks systems to build valid proofs or identify impossible ones across 70 tasks.
The starting scores are the point
No AlienCode run starts above 15.7%. That is the figure the benchmark is designed to produce: a system cannot recall its way into these worlds, because the correct answers contradict the ones it learned.
After four rounds of probing and observing, the best system reaches 89.0%. The same number of turns without feedback leaves systems between 0.5% and 11.0%. The gap between those two figures is the whole measurement: it is the value of running an experiment and reading the result, isolated from everything else the model knows.
Choosing the experiment is part of the skill
The most interesting control in the paper has systems replay probes that a different system chose. Nine of ten did worse on AlienCode that way than when they picked their own probes.
That is a result about agency rather than about knowledge. The information content of the probes was identical; what differed was whether the system had a reason for running them, and therefore whether it knew what to look for in the output.
Knowing the rule is not enough
The paper reports one more number worth keeping. Even when a system states every rule required for a task correctly, it solves the task only 73.4% of the time. Discovery and application come apart: a model can describe the mechanism it just found and still fail to use it.
Caveats
The worlds are synthetic and small, and the benchmark is published by one of the labs whose models it evaluates. The discovery targets are also fixed, which means the scores will decay in meaning as soon as the tasks reach training data — the usual fate of any benchmark whose answers are public.
What to watch
Whether an independent group runs frontier models on it and publishes the table, and whether the replay control holds up. If choosing your own experiment reliably beats being handed the same evidence, that is a measurable property worth building evaluations around.
