# AgentTrap: Measuring Runtime Trust Failures in Third-Party Agent Skills arXiv: https://arxiv.org/abs/2605.13940 ยท HTML: https://mlsanity.com/paper/2605.13940/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. - Builds on 13 of its 19 references that are in the map. - First posted on arXiv 13 May 2026. ## Placed in - LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/) ## Builds on (most cited first, 13 of 13) - [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023) - [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024) - [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024) - [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024) - [Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale](https://mlsanity.com/paper/2601.10338/) (2026) - [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://mlsanity.com/paper/2602.20156/) (2026) - ["Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild](https://mlsanity.com/paper/2602.06547/) (2026) - [MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents](https://mlsanity.com/paper/2510.15994/) (2025) - [BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoning](https://mlsanity.com/paper/2604.09378/) (2026) - [Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw](https://mlsanity.com/paper/2604.04759/) (2026) - [How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Study](https://mlsanity.com/paper/2604.03070/) (2026) - [HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?](https://mlsanity.com/paper/2604.15415/) (2026) - [Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis](https://mlsanity.com/paper/2605.00314/) (2026) ## Built on by (most cited first, 12 of 12) - [AIRGuard: Guarding Agent Actions with Runtime Authority Control](https://mlsanity.com/paper/2605.28914/) (2026) - [VIGIL: Runtime Enforcement of Behavioral Specifications in AI Agent Skills](https://mlsanity.com/paper/2606.26524/) (2026) - [Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval](https://mlsanity.com/paper/2606.10388/) (2026) - [MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection](https://mlsanity.com/paper/2608.19901/) (2026) - [EvoSkill Injection: Red-Teaming Autonomous Skill Generation and Evolution in Self-Evolving Agents](https://mlsanity.com/paper/2608.30429/) (2026) - [Versioned Transitive Dependency-Closure Binding and Operation-Time Effect Governance for Agent Skills: ClosureBound](https://mlsanity.com/paper/2609.05920/) (2026) - [From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis](https://mlsanity.com/paper/2607.24459/) (2026) - [MMSkillRisk: Can Agents Stay Safe When Multimodal Skills Become Traps?](https://mlsanity.com/paper/2609.35912/) (2026) - [Hiding in Plain Sight: Decoupling Pretext from Actuation for Skill Poisoning in LLM Agents](https://mlsanity.com/paper/2609.39352/) (2026) - [Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?](https://mlsanity.com/paper/2610.07359/) (2026) - [StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agents](https://mlsanity.com/paper/2610.17616/) (2026) - [SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models](https://mlsanity.com/paper/2610.17727/) (2026) ## Abstract (arXiv) Third-party skills are becoming the package ecosystem for LLM agents. They package natural-language instructions, helper scripts, templates, documents, and service configuration into reusable workflows. This makes skills useful, but it also introduces a new security problem: a malicious skill does not need to ask the model to perform an obviously harmful action. Instead, it can disguise the harmful behavior as part of a routine workflow, relying on the agent to execute that workflow with high-value permissions and limited human supervision. We introduce AgentTrap, a dynamic benchmark for evaluating whether LLM agents can use third-party skills while resisting malicious runtime behavior. AgentTrap contains 141 tasks: 91 malicious tasks and 50 benign utility tasks, covering 16 security-impact dimensions grounded in agent-skill supply-chain threats. In each task, the agent receives an ordinary user request, runs with installed skills that may contain malicious workflow elements, and is executed in a sandboxed environment. AgentTrap then judges complete trajectories for attack success, blocked or refused behavior, attack-not-triggered cases, and no-attack-evidence outcomes. Our central finding is that the most informative failures are not simple jailbreaks. Models often complete the visible user task while treating unsafe side effects introduced by the skill as part of the normal workflow. This motivates runtime evaluation of the concrete model--framework--workspace environment in which users actually delegate work. Code and data are available at https://github.com/zhmzm/AgentTrap and https://huggingface.co/datasets/zhmzm/AgentTrap.