A benchmark that will not let the model see the future
Most financial benchmarks test knowledge, research workflows or forecasting in isolation. A paper posted to arXiv introduces Investor’s Last Exam, or ILE, which tries to grade the whole chain instead: the evidence available at decision time, the assumptions built on it, and the judgment that follows — then scores the forecast against what actually happened.
The enforcement mechanism is the database. ILE is built on a versioned financial corpus spanning 3.43 million document versions across nine source categories, supplemented with structured market data, with validated timestamps governing both retrieval and computation. An agent asked to value a company as of a given date cannot reach a filing published after it.
The scores
Evaluation covers 86 assignments and grades research quality, forecast accuracy and simulated investment performance together. GPT-6 Astra and Claude Opus 5.5 lead overall at 68.1 and 66.9 out of 100.
The component scores are where the paper makes its argument. Even Astra, the leader, scores only 52.5 on Forecast & Model — roughly 16 points below its own overall result. The authors’ conclusion is that “even state-of-the-art models fail to translate overall research capability into predictive performance”.
These are the benchmark authors’ own measurements, not vendor-reported figures, and the paper is a preprint that has not been peer reviewed.
Built by the buy side
The benchmark’s provenance is part of its claim. ILE is described as expert-led, with experienced buy-side investors and fund managers setting the task taxonomy around professional analyst deliverables, and an evaluation framework that links each scoring criterion to its sources, calculations and assumptions through financial dependency graphs.
That last structure is what lets the benchmark say where an answer went wrong rather than only that it did. A wrong valuation can be traced to the source an agent failed to retrieve, the calculation it botched, or the assumption it asserted without support.
What the gap probably means
A model that researches well and forecasts badly is not failing at reading. It is failing at the step where evidence has to be converted into a committed number under uncertainty — the part of analyst work that has no retrievable answer because the answer has not happened yet.
Benchmarks that let a model see the outcome, even indirectly through a later document or a post-hoc summary, cannot separate the two. ILE’s frozen information set is an attempt to make that separation measurable, and the 16-point gap between overall score and forecasting score is the first number it produces.
What to watch
Whether the leaderboard holds when other groups run it, and whether the frozen-corpus design survives contact with models whose training data already covers the period under test. A timestamp gate controls retrieval; it does not control what a model memorised. That limitation applies to every benchmark of this kind, and it is the one worth asking about before treating 68.1 out of 100 as a measure of investment judgment.
