A benchmark with real defences on the other side
A paper posted to arXiv on 10 November introduces BypassGym, which its authors describe as the first benchmark to systematically test whether AI agents can evade endpoint detection and response software on real endpoints. The task set is narrow and concrete: generate and compile a loader that executes command-and-control shellcode without being blocked or terminated.
What separates it from a simulated evaluation is the environment. The agents work against live Windows systems running commercial endpoint products in virtual machines on Proxmox VE, and receive structured detection feedback after each attempt rather than a pass or fail.
The numbers
The published evaluation covers five lightweight model backends, six endpoint products and three payloads. DeepSeek-V4.1 Flash reached 17 of 18 successful runs, the highest figure reported. On the defensive side, Bitdefender allowed only 5 of 15 runs through.
The second number is the one defenders should read first. A spread that wide across commercial products, against the same agents and the same payloads, says more about the variance between endpoint tools than about any single model’s capability — and it is the kind of comparison the products’ own marketing does not produce.
The authors also run conditions with and without a supplied evasion-knowledge skill, which separates what the model knows from what it was handed. That design choice is what makes the result interpretable: a high score with the skill attached is a measurement of the scaffold, not of the model.
What it does not establish
This is a laboratory measurement of offensive capability, not a reading of what any product does in production. Scores depend on the versions and configurations tested, and three payloads is a small sample — a single product’s figure should be read as an indication rather than a ranking.
The model named at the top is also a lightweight one, which is the point the authors appear to be making: the capability does not require a frontier model.
Why publish it
Benchmarks that measure offensive capability are a recurring argument in the field, because the artefact that measures the risk is also a training target for it. The case for publishing is that defenders cannot price a risk they cannot measure, and the uneven product scores here are exactly the sort of finding that only a public benchmark surfaces.
The thing to watch is whether the endpoint vendors respond to their own numbers, and whether the authors publish the per-product breakdown in enough detail for anyone to reproduce it.
