# Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems

arXiv: https://arxiv.org/abs/2606.00925 · HTML: https://mlsanity.com/paper/2606.00925/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 66 references that are in the map.
- First posted on arXiv 30 May 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 6 of 6)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Prompt Injection Attack to Tool Selection in LLM Agents](https://mlsanity.com/paper/2504.19793/) (2025)
- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills](https://mlsanity.com/paper/2604.06550/) (2026)
- [A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks](https://mlsanity.com/paper/2509.14285/) (2025)

## Built on by (most cited first, 2 of 2)

- [SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills](https://mlsanity.com/paper/2606.15899/) (2026)
- [Understanding and mitigating the risks of OpenClaw for non-technical users: A practical guide with Skill](https://mlsanity.com/paper/2606.11007/) (2026)

## Abstract (arXiv)

Open agent platforms allow community contributors to publish reusable skills that agents can invoke at runtime. This extensibility also creates a supply-chain risk: malicious contributors can hide harmful behavior inside skills that appear benign under superficial inspection. However, existing defenses are hard to evaluate because there is no benchmark that measures both malicious-skill detection and runtime verification. We present SkillVetBench, a two-stage security vetting benchmark for open agentic skill ecosystems. The first stage performs semantic vetting over each skill's natural-language specification to detect hidden malicious intent. The second stage executes flagged skills in an instrumented sandbox to observe runtime behavior and collect auditable evidence. We build a benchmark from confirmed malicious skills in the live OpenClaw ecosystem, including samples from the recent ClawHavoc supplychain campaign. Unlike static-only methods, SkillVetBench verifies detected threats with execution traces. Our experiments show that: (1) semantic-only and signature-based baselines are insufficient, missing up to 89% of malicious skills whose threats arise from natural-language instructions, multicomponent logic, or cross-component interactions; (2) runtime attacks are concentrated in a small set of high-permission primitives, especially exec, writefile, installskill, and spawn; and (3) SkillVetBench provides case studies in which sandbox execution directly supports malicious verdicts with concrete runtime evidence.