Measuring the choice, not the output
Most evaluations of a research agent ask whether its output is any good. A preprint posted to arXiv on 29 October, rTaste: Do Research Agents Have Taste?, asks a narrower and more awkward question: does the agent know which of its own ideas is worth running, before anyone knows how it turns out?
The setup makes that measurable. Six frontier models are each given seven recent NLP papers. Working alone or as a team, each proposes minimum viable experiments, forecasts whether each will clear a threshold the model itself sets, and ranks them. Then a separate executor attempts every proposal — including the ones the proposing agent ranked last. Three model auditors judge the results.
Running the discarded experiments is the design choice that matters. Without it, a ranking cannot be scored, because nobody knows what the rejected options would have produced.
What 415 experiments produced
The agents generated 415 minimum viable experiments between them. Auditors judged 194 of those sound, valuable and new. Of that 194, fewer than half — 73 — were also judged to go beyond the seed paper and to be non-obvious.
So roughly one proposal in six clears the higher bar. That is not a damning number for an automated pipeline. The problem the paper identifies is what happens next.
The ranking does not earn its place
The agents’ first choices match the auditors’ most valued experiment more often than a random pick would. But, the authors write, those choices are not shown to improve on a far simpler rule: taking whichever experiment the agent wrote first.
The forecasts fare similarly. They beat a base rate, but are not measurably more accurate than those of a fresh instance of the same model handed only the paper and the registrations — that is, an instance with none of the context the proposing agent built up.
And teams, the configuration usually reached for when a single agent underperforms, cost more without a clear overall improvement in proposal quality.
What the result bears on
The paper’s own summary is restrained: the agents produce many worthwhile experiments, but the evidence that their rankings improve the choice of which to run remains limited. That is a claim about one capability, not about research agents in general.
The limits are worth stating. Seven NLP papers is a small and single-field seed set. The judgments of soundness, value and novelty come from model auditors rather than human reviewers, which puts a model’s taste in the hands of other models. And the design that makes the ranking measurable — running every proposal, including the rejects — is expensive enough to constrain how far the benchmark can scale.
The authors have released the evaluation harness and a dataset of all 415 proposals with their forecasts and audited outcomes. For anyone building a system that files its own research agenda, the useful finding is where to spend effort: generating candidates appears to work; deciding between them has not yet been shown to.
