The tier that never escalates is the one that matters
In two-tier triage, a fast model decides most web requests and passes the rest up to a slower, more careful tier. The design assumes the fast tier is conservative. A paper posted to arXiv on 5 November tests what happens when the fast tier meets an attack family it has never seen, and the answer is that a request the fast tier closes is never examined by anything else.
The setup holds each of seven attack families out of training, out of recalibration and out of threshold-setting, then splices the payloads into the parameters of real browsing traffic. The comparison runs zero-shot decision models — which return a probability for a typed question — against supervised detectors, a local language model, the OWASP Core Rule Set and an anomaly baseline.
A confident wrong answer
The supervised detector fails in the worst way available: not by hesitating, but by being sure. Across ten training runs, ModernBERT assigns a probability below 0.05 to between 37% and 60% of held-out path traversal requests. Within a 5% benign budget it escalates only 50% to 81% of them, against nearly all of the same requests once traversal is in its training data.
The gap is not a calibration wobble. The same requests, judged by the same model, move from almost always escalated to confidently closed depending on whether the family appeared in training.
The hosted models do better, with a caveat the authors state
Jev and Clef, two hosted decision models whose budget decisions do not depend on the held-out family, escalate at least 97.6% of every family within the same budget. The authors immediately qualify it: the training data of both is undisclosed, and these are public attack families, so the held-out condition may not be held out for them at all.
They also report the cost of that caution. Jev escalates 37% of benign twins — requests built from the exploits with the attack replaced by legitimate values — which is a false-positive rate on traffic that looks like an attack and is not. And detectors trained on older exploits turn out to have learned what scanner requests look like rather than what an attack is.
Findable, and unstable
The last result is the one to sit with. A search guided by Jev’s own probability gets 43% of held-out attacks past Jev within 300 queries. That is a practical attack on the triage layer, using the defender’s score as the signal.
What complicates it is that Jev does not answer the same way twice: asked again, it escalates those same requests in 60% of its answers. A defender reading that as robustness is misreading it — the variance means an attacker only needs the request to be closed once, while the defender needs it escalated every time.
The practical reading for anyone running two-tier triage is that measuring a fast tier on known families overstates what it will do on the next one, and that a per-request confidence score is both the thing the system runs on and the thing an attacker optimises against.
