A benchmark where the answer already exists
Most agent benchmarks for knowledge work are generative — build a model, a memo, a spreadsheet from a brief, then have a rubric or another model judge it. kw-bench, posted to arXiv on 26 October by George Juraj Salapa, goes after the other half of the job: the investigative part, where the answer already exists and has to be found.
Its first task family is a five-year VAT reconstruction spanning three generations of SAP — legacy ECC, a partial S/4HANA migration and a consultant’s redesign. The paper says the data is delivered as raw system extracts: page headers, local number formats, partial link tables, and no documentation of how any of it connects. Grading is deterministic. There is no model acting as judge.
The result that makes it a story
The author ran four harnesses — Claude Code, Codex, OpenCode and micro-cc — on the same model, DeepSeek v4.1 Flash, for two hours each.
The two model-agnostic harnesses produced partial answers: OpenCode scored 0.542 and micro-cc 0.528. The two lab harnesses, running a model other than the one they were built around, scored 0.00 each. The paper attributes the failure to inertia — the difficulty of keeping many weak signals alive across a long session — rather than to any single wrong step.
On a frontier model the task stops being hard. micro-cc and Claude Code running Claude Opus 5.5 scored 0.9986 and 1.000, in 22 and 25 minutes respectively. The paper’s summary of that gap is the line worth quoting: below the frontier, the harness decides how far a model gets.
What it does and does not establish
The claim is narrower than the headline number suggests, and the paper is upfront about its scope. This is one task family, from one author, with one model standing in for everything below the frontier and one standing in for the frontier. A harness scoring zero on a VAT reconstruction with a mid-tier model is not evidence that it scores zero generally.
What the design does do well is remove the usual escape hatches. Deterministic grading means there is no judge model to flatter a plausible-sounding answer, and an engagement-derived task means the failure modes are the ones specialists actually hit: inconsistent formats, missing join keys, no schema.
Why the harness question matters commercially
The practical implication runs against how agent products are usually sold. Vendors market the model; buyers increasingly mix models. If the scaffolding around the model carries as much of the result as this benchmark suggests at the mid tier, then a harness tuned to one lab’s model is a different product when pointed at another lab’s.
The paper also makes a cost claim that is easy to check and worth checking: that a weeks-long engagement of this type now costs minutes and a few dollars. The code and tasks are published, which means the zero scores are reproducible by anyone who disagrees with them. That is the next thing to watch — whether the two harnesses that scored nothing score nothing when someone else runs them.
