# MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

arXiv: https://arxiv.org/abs/2608.19901 · HTML: https://mlsanity.com/paper/2608.19901/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 23 of 23)

- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale](https://mlsanity.com/paper/2601.10338/) (2026)
- [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://mlsanity.com/paper/2602.20156/) (2026)
- [MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers](https://mlsanity.com/paper/2508.14925/) (2025)
- [Formal Analysis and Supply Chain Security for Agentic AI Skills](https://mlsanity.com/paper/2603.00195/) (2026)
- [SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems](https://mlsanity.com/paper/2604.06811/) (2026)
- [How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Study](https://mlsanity.com/paper/2604.03070/) (2026)
- [SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces](https://mlsanity.com/paper/2605.12015/) (2026)
- [MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills](https://mlsanity.com/paper/2606.07131/) (2026)
- [When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems](https://mlsanity.com/paper/2606.00448/) (2026)
- [AgentTrap: Measuring Runtime Trust Failures in Third-Party Agent Skills](https://mlsanity.com/paper/2605.13940/) (2026)
- [Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malware](https://mlsanity.com/paper/2607.02357/) (2026)
- [Detecting Malicious Agent Skills in the Wild using Attention](https://mlsanity.com/paper/2606.23416/) (2026)
- [RouteGuard: Internal-Signal Detection of Skill Poisoning in LLM Agents](https://mlsanity.com/paper/2604.22888/) (2026)
- [ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree](https://mlsanity.com/paper/2606.01494/) (2026)
- [SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction](https://mlsanity.com/paper/2606.02540/) (2026)
- [Seeing Is Not Screening: Multimodal Hidden Instruction Attacks on Agent Skill Scanners](https://mlsanity.com/paper/2606.18198/) (2026)
- [ColluSkill: Adversarial Cross-Skill Composition for Evading Agent Skill Scanners](https://mlsanity.com/paper/2608.09732/) (2026)
- [Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security](https://mlsanity.com/paper/2606.11671/) (2026)
- [SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents](https://mlsanity.com/paper/2607.25619/) (2026)

## Built on by (most cited first, 1 of 1)

- [SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models](https://mlsanity.com/paper/2610.17727/) (2026)

## Abstract (arXiv)

Agent Skills extend LLM agents with reusable instruction packages that may also include scripts, resources, and service configuration. This creates a direct distribution channel for malicious behavior, yet existing malicious-Skill datasets are fragmented across sources, artifact formats, evidence regimes, and benign coverage; duplicated and structurally related content further complicates direct aggregation and evaluation. We present MaliciousSkillBench, a comprehensive benchmark for malicious Agent Skill detection. We consolidate 13 public sources, 11 of which contribute Core malicious artifacts, and reduce 8,414 raw malicious records to 7,539 normalized-unique identities in 4,588 operational structural families. After conservative cross-label conflict exclusion, the primary benchmark contains 9,740 Skills: 7,505 malicious and 2,235 benign. To characterize its coverage, we harmonize 11 attack categories for 4,983 malicious identities with supported source-native mappings and find substantial differences in threat composition across sources. We then evaluate three learned text detectors and three off-the-shelf Skill scanners. Learned detectors achieve 0.882-0.932 Random Macro-F1 but only 0.653-0.665 under Source-Disjoint evaluation; the strongest word TF-IDF SVM scores 0.932/0.916/0.665 on Random/structural-disjoint/Source-Disjoint while retaining 95.6% malicious recall but producing 62.4% benign FPR on held-out sources. Off-the-shelf scanners occupy different but also unsatisfactory operating regimes, reducing false positives only at the cost of sharply lower malicious recall. Together, these results show that reliable malicious-Skill detection requires both broader cross-source benchmark coverage and evaluation that jointly measures attack detection and benign over-flagging.