Letting the model build its own scaffolding
Jens Lehmann and Sahar Vahdati have published a controlled study on arXiv asking whether a frontier model can write and then improve the agent harness that drives it — the code that orchestrates model calls, manages state and decides what to do next. They posted it on 21 October.
The setup is deliberately clean. Each model builds a harness, tests it on five development games from ARC-AGI-3, and revises it twice. Performance is then measured on twenty public games held out entirely from construction and revision. The scoring metric, Relative Human Action Efficiency, combines level completion with action efficiency against hidden human baselines: finishing a level within five times the human action count earns a per-level score, capped at 115, with later levels weighted more.
Nine model-and-effort conditions were run across seven model families, three independent constructions each, for 27 lineages in total. The families include GPT-6 Astra, four GPT-5.6 variants, Claude Fable 5, Claude Opus 5 and Claude Sonnet 5.
The number that includes zero
After two revisions, mean held-out score rose by 2.80 points, with a 95% bootstrap interval of −1.44 to 7.35. The interval contains zero, so the study cannot distinguish the gain from noise. Fifteen lineages improved and twelve got worse, a split a two-sided sign test puts at p = 0.70.
Mean development score rose by 8.42 points over the same two revisions. The difference between the two — 5.62 points, interval 1.78 to 9.77 — is the part the authors call the adaptation gap, and it is the result that does separate from zero. Revising a harness against five games reliably improves it on those five games, and that improvement does not transfer.
Initial held-out scores averaged 24.54 and ranged from 0 to 97.14, which is the other thing worth noting: the variance between lineages dwarfs the variance from revising any one of them.
Cheaper, not better
Cost moved more than capability. Mean cost to solve a held-out game fell from $10.30 initially to $7.15 after one revision and $7.57 after two. But the median change was only 54 cents, against a mean drop of $2.72 — a few lineages found large savings and most found none.
Of the 27 revision-2 descendants, nine improved both score and cost, nine got cheaper while losing score, six gained score at higher cost, and three got worse on both. The whole experimental programme cost at least $18,129.27 under the paper’s own price schedule.
Against the hand-built systems
The authors set their results beside human-steered harnesses reported elsewhere, with the caveat that those systems used different development and selection procedures, so the comparison does not isolate the effect of human steering.
The numbers are still stark. Tycho reaches 100 on all public games with two models, VISTA and AVO reach 100 with Opus 5, and Retrodict reaches 99.86. A generic Claude Code baseline scores 42.83 against 98.98 for the same project’s purpose-built world-model harness. The best autonomously built lineage here reached 97.14; the typical one started near 24.
What to watch
The honest reading is that autonomous harness construction works sometimes and the study cannot tell you in advance which time. That is a different claim from “self-improvement does not work”, and the authors do not make the stronger one.
What it does establish is a methodological trap worth naming. A development set of five games gives a clear, reproducible improvement signal that mostly measures fit to those five games. Anyone reporting a self-improving agent without a held-out set is reporting the 8.42, not the 2.80.
