A new benchmark asks coding agents to rebuild a working app they can only interact with
GPT-6 Astra managed 49.2% on the hardest setting and Claude Opus 5 28.8%, in the paper's own evaluation of nine frontier coding agents.
Benchmark
ProgramDistill
Whether a coding agent can rebuild the behaviour of a working interactive web application after parts of it have been removed, using the running reference app rather than a written specification as the statement of what the code should do.
- Maintainer
- Microsoft Research
- Score
- %
- Last checked
- September 19, 2026
Leaderboard
No verified results recorded yet.
Methodology
An automated pipeline called mine-craft-patch interacts with 26 fully functional web applications, records 1,975 replay-verified behaviours and turns them into 4,063 tasks spanning atomic repair, cumulative repair and full-application reconstruction. A task counts as solved when the agent's code reproduces the recorded behaviour on replay. Difficulty is controlled by restoration depth — how many layers of the application were stripped out before the agent starts.
Important limitations
The benchmark covers interactive web applications only, so it says nothing about systems programming, data engineering or non-interactive code. Tasks are generated without human review, and the reported figures come from the paper's own evaluation of nine coding agents rather than from an independently run leaderboard.