A testbed for fraud that does not disappear
Researchers have published ScamWorld, an interactive environment that reconstructs reported scam websites so that web agents can be tested against them repeatedly. The problem it is built to solve is mundane: real scam sites are short-lived, which makes any systematic evaluation of how agents behave on them difficult to run twice.
Each reconstructed site reproduces a documented fraud mechanism — the paper names fraudulent online stores, fake investment platforms and advance-fee schemes — across different scenarios and languages. Crucially, the signs of fraud are placed where the agent can observe them before it takes its final action. An agent that proceeds is not being ambushed; it is proceeding past evidence it could have read.
Between one in nine and one in three
Across web agents powered by six frontier large language models, the authors report baseline deception rates ranging from 11% to 35%. That is the share of runs in which the agent completed the fraudulent action while carrying out an ordinary user task. The figures are the authors’ own and have not been independently reproduced.
The spread matters as much as the range. A fourfold difference between the best and worst model on the same reconstructed sites suggests the behaviour is not a fixed property of agents as a category, and that whatever separates the careful models from the credulous ones is something a developer can affect.
Payments go through more easily than passwords
The second finding is about what the site asks for. Agents proceeded more readily with payments than with requests for credentials or wallet information, the paper reports. That ordering is not obviously the safe one. A handed-over password is recoverable in a way a completed payment to a fraudulent merchant generally is not, and a refusal reflex trained on “never share your login” does not transfer to “do not buy from this shop”.
Some agents saw it and continued
The most uncomfortable result comes from reading the trajectories rather than the scores. The authors found that some agents acknowledged suspicious conditions and proceeded anyway, at times treating the user’s instructions as authorisation to accept additional terms the website imposed.
That is a different failure from not noticing. An agent told to buy a particular item appears, in these runs, to read the instruction as a mandate that survives contact with evidence the shop is fraudulent. The user authorised a purchase; the agent treated that as authorising this purchase, on these terms.
What to watch
ScamWorld is offered as a reusable testbed, which means the number that matters next is whether vendors report against it and whether deception rates move when they do. The harder question the paper raises is one of interface rather than capability: an agent that correctly identifies a scam and completes the transaction anyway does not need a better detector. It needs permission to stop.
