Grading the workflow, not just the answer
Most evaluations of AI on financial work ask whether the model can pull the right number out of a document and compute with it correctly. A benchmark posted to arXiv on 7 November argues that this misses what the job actually consists of: choosing between conflicting sources, making defensible assumptions, and keeping every step auditable.
DeepFinance is built around 21 end-to-end private equity and investment banking tasks that take a human professional an average of 27.1 hours. Each pairs a request with a data room averaging 19 files, and is graded against an average of 84 expert-written rubric criteria covering output, workflow and judgement. Twenty-two model configurations were evaluated, each inside its own native agent harness.
The best configuration, GPT-6 Astra at maximum reasoning, satisfied 57.6% of rubric criteria. The authors note that the top four configurations are not statistically distinguishable from one another — with 21 tasks, the leaderboard’s ordering at the top carries less information than its headline number.
Correct arithmetic on the wrong inputs
The sentence in the paper that does the most work is a description of failure, not a score. Frontier models, the authors write, often calculate correctly from the wrong inputs, and find decision-relevant evidence without acting on it.
Both halves of that matter in different ways. The first is the harder problem: a model that computes accurately from a figure it should not have used produces an answer that survives every arithmetic check and is still wrong. There is no internal inconsistency for a reviewer to catch. The second is in some ways more surprising — the evidence was located, it was relevant to the decision, and the model carried on without incorporating it.
Why a rubric rather than an answer key
Work of this shape has no single correct output. The rubric is how the authors make it gradable: 84 criteria per task, written by practitioners, each either met or not. That buys auditability, and it costs precision in a specific way. A percentage here counts criteria satisfied, not whether the resulting investment judgement would have been sound. A model could satisfy most of a rubric and still reach a recommendation a partner would reject.
The 27.1-hour average is the other number worth holding on to. These are not prompts; they are multi-day pieces of work compressed into a single evaluation, which is why the set is only 21 tasks and why the statistical caveat about the top four configurations follows from the design rather than from carelessness.
What it says about deploying agents on deals
The practical reading is narrower than “AI cannot do diligence”. Satisfying 57.6% of a professional rubric on 27-hour tasks is substantial, and the gap between the leaders is not meaningful at this sample size. The caution is about the shape of what remains: the errors are not sloppy arithmetic, which review catches, but sound arithmetic on inputs that should have been questioned, which review often does not.
The figures are the authors’ own and have not been independently reproduced. What would make the benchmark more useful is more tasks — enough to separate the top configurations — and a second pass on whether the models that find decision-relevant evidence and ignore it do so for reasons a prompt can fix.
