# MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills

arXiv: https://arxiv.org/abs/2606.07131 · HTML: https://mlsanity.com/paper/2606.07131/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks](https://mlsanity.com/paper/2504.11358/) (2025)
- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale](https://mlsanity.com/paper/2601.10338/) (2026)
- [Attention Tracker: Detecting Prompt Injection Attacks in LLMs](https://mlsanity.com/paper/2411.00348/) (2024)
- [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://mlsanity.com/paper/2602.20156/) (2026)
- [MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents](https://mlsanity.com/paper/2502.05174/) (2025)
- [Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks](https://mlsanity.com/paper/2005.09535/) (2020)
- [Taxonomy of Attacks on Open-Source Software Supply Chains](https://mlsanity.com/paper/2204.04008/) (2022)
- [Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats](https://mlsanity.com/paper/2603.11619/) (2026)
- [The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey](https://mlsanity.com/paper/2603.11088/) (2026)
- [Formal Analysis and Supply Chain Security for Agentic AI Skills](https://mlsanity.com/paper/2603.00195/) (2026)
- [SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces](https://mlsanity.com/paper/2605.12015/) (2026)
- [Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis](https://mlsanity.com/paper/2605.00314/) (2026)
- [On the Feasibility of Cross-Language Detection of Malicious Packages in npm and PyPI](https://mlsanity.com/paper/2310.09571/) (2023)
- [MalGuard: Towards Real-Time, Accurate, and Actionable Detection of Malicious Packages in PyPI Ecosystem](https://mlsanity.com/paper/2506.14466/) (2025)
- [No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills](https://mlsanity.com/paper/2605.13044/) (2026)

## Built on by (most cited first, 19 of 19)

- [Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malware](https://mlsanity.com/paper/2607.02357/) (2026)
- [Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval](https://mlsanity.com/paper/2606.10388/) (2026)
- [ColluSkill: Adversarial Cross-Skill Composition for Evading Agent Skill Scanners](https://mlsanity.com/paper/2608.09732/) (2026)
- [Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents](https://mlsanity.com/paper/2608.12851/) (2026)
- [AgentFlow: Building Agent Dependency Graphs for Static Analysis of Agent Programs](https://mlsanity.com/paper/2607.01640/) (2026)
- [SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignment](https://mlsanity.com/paper/2608.07639/) (2026)
- [MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection](https://mlsanity.com/paper/2608.19901/) (2026)
- [Versioned Transitive Dependency-Closure Binding and Operation-Time Effect Governance for Agent Skills: ClosureBound](https://mlsanity.com/paper/2609.05920/) (2026)
- [Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents](https://mlsanity.com/paper/2607.12340/) (2026)
- [SkillSentry: Adaptive Honey Worlds for Dynamic Safety Testing of Agent Skills](https://mlsanity.com/paper/2608.03485/) (2026)
- [SkillSpec: Intent-Masked Specification Reasoning for Agent Skill Correctness](https://mlsanity.com/paper/2609.06052/) (2026)
- [SkillSecurer: Detecting and Patching Prompt-Injection Vulnerabilities in AI Agent Skills](https://mlsanity.com/paper/2609.14079/) (2026)
- [MMSkillRisk: Can Agents Stay Safe When Multimodal Skills Become Traps?](https://mlsanity.com/paper/2609.35912/) (2026)
- [SKILLLITE: Evidence-Guided Malicious Skill Auditing with Compact LLMs](https://mlsanity.com/paper/2609.36879/) (2026)
- [Runaway Reaction: When Benign Skills Compose into Malicious Behavior](https://mlsanity.com/paper/2610.05943/) (2026)
- [SpecGuard: Proving a Task Is Broken Before the Agent Cheats](https://mlsanity.com/paper/2610.09159/) (2026)
- [SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models](https://mlsanity.com/paper/2610.17727/) (2026)
- [Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution](https://mlsanity.com/paper/2610.18927/) (2026)
- [CoRedSkill: Stealthy Skill Composition Attacks via AST-Guided Injection](https://mlsanity.com/paper/2610.20049/) (2026)

## Abstract (arXiv)

AI coding agents such as Claude Code and Gemini CLI increasingly extend themselves with third-party skills: markdown packages bundling natural-language instructions, executable scripts, and tool permissions. Because a skill is at once code and agent-facing instruction, it introduces a supply chain dependency whose risk is neither pure code nor pure prompt. Detection tools have never been measured against verified ground truth spanning this hybrid space, leaving their effectiveness unknown and wild-only evaluations biased. We present MalSkillBench, the first runtime-verified benchmark of malicious agent skills: 3,944 malicious skills labeled along a three-dimensional taxonomy of 108 cells. Of these, 3,214 come from a closed-loop Generate-Verify-Feedback pipeline admitting only samples whose malicious behavior fires inside a Docker sandbox under system-call monitoring and an LLM judge; we add 703 in-the-wild and 4,000 matched benign skills. Our measurements are consistent: code injection reaches 94.5% verification yield but prompt injection only 75.8%, the same fragility that later makes it hard to detect; the wild sample is narrow, dominated by one cryptocurrency-theft campaign (86.6% one behavior, 81% from two accounts) with a small but architecturally new tail attacking the agent control plane; the strongest skill-specific detector reaches 98.4% recall on code injection yet collapses on prompt-injection and agent-control attacks, and wild-only scoring swings the ranking by up to 66 recall points; supply-chain scanners and prompt-injection defenses each see only half of a skill, and no combination recovers the code-instruction relationship. Detecting malicious skills therefore requires reasoning jointly over task intent, code, and instructions. We release the dataset, pipeline, baselines, and results.