A benchmark that runs against real servers
Researchers at Peking University’s Zhongguancun Academy, Meituan and the Hong Kong University of Science and Technology have published MCPHallu, a benchmark for the failures that make a tool-using agent report something that did not happen. It went up on arXiv on 5 November.
The design choice that matters is that the tools are real. The 358 tasks run against 36 production Model Context Protocol servers inside isolated containers, not against mocked responses. Each task is built around a structural trigger for one of four failure types: committing to a single branch of a plan too early, pursuing a goal the available tools cannot reach, calling a tool in a way it was not meant for, and losing earlier context.
The scores
Fourteen agents from eight vendors were run over 5,012 trajectories and scored from 0 to 1, higher being better. The average across all of them is 0.642.
Claude Opus 4.6 leads at 0.728, followed by Claude Sonnet 4.5 at 0.710, Claude Sonnet 4.6 at 0.702 and Claude Opus 4.5 at 0.696. ByteDance’s Doubao-Seed-1.8 is next at 0.692, then GPT-5-chat at 0.659 and DeepSeek-V3.1 at 0.655. At the bottom are Gemini 3 Flash at 0.583 and GLM-5.1 at 0.539.
One failure type does the damage
The overall column hides the result worth reporting. Averaged across the 14 agents, Tool Misuse is the easiest subtype and Unreachable Goal is the hardest — it is the worst subtype for 12 of the 14 models.
The gap is wide. On Tool Misuse, Claude Opus 4.6 scores 0.885 and even GLM-5.1 manages 0.688. On Unreachable Goal, the best score in the table is Claude Sonnet 4.6’s 0.585, DeepSeek-V4 falls to 0.302 and Gemini 3 Flash to 0.284. Across the set, agents passed 23.2% of Unreachable Goal runs, for a subtype average of 0.458.
In other words, these agents are considerably better at using a tool correctly than at saying that the tools they have cannot do what was asked. The paper also reports an average rank shift of 6.9 out of 13 positions between subtypes, and a correlation of −0.12 between Branch Collapse and Unreachable Goal scores across models — being good at one says close to nothing about the other.
What it does not settle
The benchmark is run by its own authors, and the line-up is a generation behind in places: OpenAI is represented by GPT-5-chat and GPT-4o, Anthropic by Claude 4.5 and 4.6 rather than its current releases. A rerun on today’s models would be a different table.
The structural finding is more durable than the ranking. If an overall score moves an agent 6.9 places depending on which failure you measure, a single number is the wrong way to choose one — and the number that matters for a deployment is usually the one for the failure that deployment cannot tolerate.
