One significant result out of four
Vals AI published the test on 9 October. It built 50 complete web applications from written product specs, deployed each with Docker Compose, then set browser agents to click through the workflows and scored each build on the share of interface tests that passed.
Two models ran it: GPT 6 Sol and Claude Opus 5.5. Each ran twice over — once as a single agent, once as a lead agent delegating to subagents — at medium and at maximum reasoning effort. That is eight configurations, each run once.
The summary Vals AI gives is blunt: “Teams cost 1.8 to 5.1 times as much as single agents, and only one difference in score was statistically significant.”
The one that was significant belonged to the weaker configuration. GPT 6 Sol at medium effort scored 77.6% alone and 84.9% as a team, a gain of 7.3 points at p = 0.005. The other three comparisons returned p-values between 0.16 and 0.81. Sol at maximum effort went from 89.0% to 90.4%. Claude Opus 5.5 at medium effort went from 91.5% to 91.2% — slightly down. At maximum effort it went from 89.8% to 93.2%, the largest raw gain in the table and still not significant on a single run.
What the teams cost
The multipliers vary by model and setting rather than tracking the benefit. Teams cost 2.5 times a single agent for Sol at medium effort, 1.8 times at maximum, 2.2 times for Opus at medium — and 5.1 times for Opus at maximum effort, the configuration whose gain Vals AI could not call significant.
Wall-clock time moved in opposite directions for the two models. Running subagents in parallel shortened Sol’s runs. Opus’s teams ran up to 2.3 times longer than its single agents, so for that model the delegation bought neither a reliable score nor speed.
The cheaper lever was already there
The more useful number in the study is not about teams at all. For Sol, turning reasoning effort up to maximum lifted the single agent by 11.4 points, at p < 0.001 — a bigger, better-established gain than the one its medium-effort team produced. For Opus, raising effort alone did not significantly change the score either.
Put together, that is an argument about where the spare budget should go. On this benchmark, for one of the two models, the knob that already exists outperformed the architecture that costs between 1.8 and 5.1 times more to run.
What to watch
The limits are the study’s own and Vals AI states them: each of the eight configurations was run once, so single comparisons carry wide uncertainty, and three of four differences cannot be separated from noise on that evidence. Passing a browser agent’s click-through is also not the same as an application being correct or secure.
What the result does bear on is a claim the industry has been making in product announcements rather than in published measurements — that delegating to more agents makes the output better. On the one benchmark where somebody checked both the scores and the bill, it mostly made it more expensive. The thing to watch is whether any vendor publishes a controlled comparison of its own, with the token cost next to the score.
