# SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces

arXiv: https://arxiv.org/abs/2605.12015 · HTML: https://mlsanity.com/paper/2605.12015/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 36 of its 70 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 30 of 36)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025)
- [WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks](https://mlsanity.com/paper/2504.18575/) (2025)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [Inducing Programmatic Skills for Agentic Tasks](https://mlsanity.com/paper/2504.06821/) (2025)
- [Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale](https://mlsanity.com/paper/2601.10338/) (2026)
- [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://mlsanity.com/paper/2602.20156/) (2026)
- [GTA: A Benchmark for General Tool Agents](https://mlsanity.com/paper/2407.08713/) (2024)
- [SkillX: Automatically Constructing Skill Knowledge Bases for Agents](https://mlsanity.com/paper/2604.04804/) (2026)

## Built on by (most cited first, 19 of 19)

- [MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills](https://mlsanity.com/paper/2606.07131/) (2026)
- [SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction](https://mlsanity.com/paper/2606.02540/) (2026)
- [ColluSkill: Adversarial Cross-Skill Composition for Evading Agent Skill Scanners](https://mlsanity.com/paper/2608.09732/) (2026)
- [AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents](https://mlsanity.com/paper/2607.27294/) (2026)
- [HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses](https://mlsanity.com/paper/2608.06984/) (2026)
- [Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents](https://mlsanity.com/paper/2608.12851/) (2026)
- [Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach](https://mlsanity.com/paper/2607.10534/) (2026)
- [MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection](https://mlsanity.com/paper/2608.19901/) (2026)
- [OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution](https://mlsanity.com/paper/2608.00677/) (2026)
- [TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation](https://mlsanity.com/paper/2608.17588/) (2026)
- [ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents](https://mlsanity.com/paper/2608.21101/) (2026)
- [Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Security](https://mlsanity.com/paper/2608.29596/) (2026)
- [Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents](https://mlsanity.com/paper/2609.01487/) (2026)
- ["You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused](https://mlsanity.com/paper/2609.32616/) (2026)
- [Trust the Brand, Lose Control: How Identity Hijacks LLM Agent Orchestration](https://mlsanity.com/paper/2609.32635/) (2026)
- [MMSkillRisk: Can Agents Stay Safe When Multimodal Skills Become Traps?](https://mlsanity.com/paper/2609.35912/) (2026)
- [Hiding in Plain Sight: Decoupling Pretext from Actuation for Skill Poisoning in LLM Agents](https://mlsanity.com/paper/2609.39352/) (2026)
- [APEX: Active Protection at Execution Boundaries for LLM Agents](https://mlsanity.com/paper/2610.06966/) (2026)
- [SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models](https://mlsanity.com/paper/2610.17727/) (2026)

## Abstract (arXiv)

Reusable skills are becoming a common interface for extending large language model agents, packaging procedural guidance with access to files, tools, memory, and execution environments. However, this modularity introduces attack surfaces that are largely missed by existing safety evaluations: even when the user request is benign, unsafe influence may reside in skill guidance, local artifacts, or execution-environment files that steer the agent toward unsafe actions. We present SkillSafetyBench, a runnable benchmark for evaluating such skill-facing safety failures. SkillSafetyBench includes 155 adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories, each evaluated with a case-specific rule-based verifier. Experiments with multiple CLI agents and model backends show that non-user attacks can consistently induce unsafe behavior, with distinct failure patterns across domains, attack methods, and scaffold-model pairings. Our findings suggest that agent safety depends not only on model-level alignment, but also on how agents interpret skills, trust workflow context, and act through executable environments. The complete benchmark is available at https://github.com/AI45Lab/skill-safety-bench.