One task separates the top of the board

Of the 56 research-level formalisation tasks on Autoform-Hard, 48 are solved by both Claude Fable 5.2 and GPT-6.2 Sol. GPT-6.2 Astra solves those 48 and one more, and no other model on the board solves it.

The extra task is a 2006 theorem of Agata Smoktunowicz about how fast a graded algebra can grow. Vals AI, which runs the board, says in a post published on 18 November that the resulting Lean proof is “the only formalization we’ve ever heard of” for that theorem.

What the theorem says, and what the run cost

The statement is a gap result. A finitely generated algebra over a field that is graded by the natural numbers, has no zero divisors and has a finite-dimensional degree-zero part has Gelfand–Kirillov dimension either at most 2 or at least 3 — never strictly in between.

Interlocking gears and a toothed belt photographed close up
The proof takes a different route from the published argument. Illustrative stock image. Rafael Minguet Delgado · pexels · Pexels License

The model worked alone in a sandbox with Lean, Mathlib and a shell. The proof runs to 1,794 lines of Lean, took 94 minutes of wall clock and 146 agent turns, and cost $21.58 at OpenAI’s list price. Autoform-Hard allows three hours and a $200 cap per task.

It nearly did not happen. About 47 minutes in, the model submitted a proof that was complete for the commutative case but left a sorry in the general one. The grader rejected it, and the model filed an honest report that the task was unsolved. The harness does not accept a give-up until the agent has spent $20; the model had spent about $8, so it was sent back. It spent nine minutes thinking without writing code, started a different approach, and the grader accepted the full proof 39 minutes after the “unsolved” report.

How the result was checked

Autoform-Hard grades in a clean room. The theorem is rebuilt from a trusted copy of the task, the statement has to match the original exactly, the benchmark’s definitions must be unchanged, and the proof may use only Lean’s standard axioms. An independent kernel then replays it. Vals also reviewed this proof by hand and reports no sorry, no native_decide, no new axioms and no redefined benchmark objects.

The proof is not a formalisation of Smoktunowicz’s argument. After a shared opening it goes its own way, around three deep results from algebra and algebraic geometry that the published proof leans on and that Mathlib does not contain. In their place the model defined a quantity it calls “capacity” — how much a space grows when multiplied by a fixed generating piece — and built the rest on that.

Rows of bound volumes on dark wooden library shelves
The published proof relies on three results Mathlib does not contain. Illustrative stock image. Oleksandra Zelena · pexels · Pexels License

What the board looks like

On the Autoform-Hard leaderboard, GPT-6.2 Astra sits first at 87.50%, at $15.24 per task. Claude Fable 5.2 and GPT-6.2 Sol tie at 85.71%, at $23.21 and $8.09 per task respectively. Gemini 4 Carbon follows at 71.43%, Gemini 4 Hydrogen at 64.29% and Gemini 4 Argon at 48.21%. Beam 1.0 and MiniMax-M3.1-Flash score zero.

Fifty-six tasks is a small board, and the tasks were chosen precisely because none of the ten frontier models in the original AutoformBench study solved them. One task moves a score by 1.8 points, which is exactly the margin separating first place from second. The underlying benchmark is Meta FAIR’s; Vals corrected five of the tasks, three of which were false as written.