Malicious intent spread across four places
Agentic systems increasingly run on reusable skills: packages that combine executable code, natural-language instructions, configuration artifacts and external tool integrations. That mix is a software supply-chain problem of a specific shape, because intent can be distributed across the instructions, the code, the metadata and the runtime behaviour, leaving any single alert hard to interpret on its own.
A paper posted to arXiv on 7 November evaluates how well current tools cope. The authors tested three open-source scanners on a held-out set of 140 skills: 80 malicious and 60 benign.
High recall, and a wall of false alarms
Cisco’s Skill Scanner in its default configuration achieved a recall of 0.91 — it caught most of the malicious skills — while producing 33 false positives among the 60 benign ones. Turning on its optional LLM analyzer, running Qwen3:8B, made the trade-off worse rather than better: recall fell to 0.85 and false positives rose to 40.
Those are the numbers worth sitting with. A scanner that flags more than half of the harmless skills in a test set is not a filter; it is a queue of work for whoever has to triage it, and the predictable outcome is that the alerts get ignored.
What the sandbox adds
The authors then propose a framework that combines static findings, safety-constrained sandbox evidence and LLM-based reasoning, tracking where each finding came from and relating evidence across the analysis layers.
Their ablation holds the reasoning backend fixed and varies only the evidence. Reasoning over the skill’s content alone gives an F1 of 0.70. Adding dynamic evidence from the sandbox raises it to 0.85. Combining static and dynamic evidence takes it to 0.89, with five false positives among the 60 benign skills. Dynamic evidence, they report, provides the largest observed improvement — watching what a skill does, rather than reading what it says, is where the signal is.
The full pipeline, and the limits of the comparison
The complete pipeline detects 68 of the 80 malicious skills, for 0.88 accuracy, 0.93 precision and 0.85 recall. Set against the LLM-enabled Cisco configuration, it matches the observed recall while reducing false positives from 40 to five.
The authors are explicit about what that comparison does not establish: it is a system-level result and does not isolate model or architecture effects. The two systems differ in more than one way at once, so the eightfold reduction in false alarms is attributable to the pipeline as a whole rather than to the sandbox specifically — even though the internal ablation points at dynamic evidence as the main contributor.
With 140 skills, these are indicative rates rather than precise ones, and the figures are the authors’ own. The question the paper leaves open is how any of this holds against skills written to evade a sandbox, which is the natural next move once dynamic analysis becomes the thing that catches you.
