mlsanityData to 7 Oct 2026

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

arXiv 2406.13352 ↗First posted 19 Jun 2024cs.CR cs.LGcited by 129

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

AI agents aim to solve complex tasks by combining text-based reasoning with external tool calls. Unfortunately, AI agents are vulnerable to prompt injection attacks where data returned by external tools hijacks the agent to execute malicious tasks. To measure the adversarial robustness of AI agents, we introduce AgentDojo, an evaluation framework for agents that execute tools over untrusted data. To capture the evolving nature of attacks and defenses, AgentDojo is not a static test suite, but rather an extensible environment for designing and evaluating new agent tasks, defenses, and adaptive attacks. We populate the environment with 97 realistic tasks (e.g., managing an email client, navigating an e-banking website, or making travel bookings), 629 security test cases, and various attack and defense paradigms from the literature. We find that AgentDojo poses a challenge for both attacks and defenses: state-of-the-art LLMs fail at many tasks (even in the absence of attacks), and existing prompt injection attacks break some security properties but not all. We hope that AgentDojo can foster research on new design principles for AI agents that solve common tasks in a reliable and robust manner.. We release the code for AgentDojo at https://github.com/ethz-spylab/agentdojo.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 51 of its 71 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

51 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  2. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
37 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  10. 2022A Generalist Agentcited by 802
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  12. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  13. 2022PAL: Program-aided Language Modelscited by 635
  14. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  15. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  16. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  17. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  18. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  19. 2023Language Models can Solve Computer Taskscited by 371
  20. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  21. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  22. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  23. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  24. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  25. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  26. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  27. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  28. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  29. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  30. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  31. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  32. 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 79
  33. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
  34. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  35. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40
  36. 2024Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competitioncited by 20
  37. 2019A critique of the DeepSec Platform for Security Analysis of Deep Learning Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 55; 2026: 64.

129 papers.

Sort:

By area:

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  2. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  3. 2025Defeating Prompt Injections by Designcited by 256
  4. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  5. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 144
  6. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  7. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  8. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  11. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  12. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
38 more
  1. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 89
  2. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 87
  3. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 67
  4. 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakagecited by 64
  5. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  6. 2025MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Serverscited by 63
  7. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 49
  8. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  9. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
  10. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  11. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agentscited by 42
  12. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 37
  13. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  14. 2025Firewalls to Secure Dynamic LLM Agentic Networkscited by 34
  15. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  16. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  17. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
  18. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  19. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  20. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  21. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 25
  22. 2026"Humans welcome to observe": A First Look at the Agent Social Network Moltbookcited by 25
  23. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injectioncited by 25
  24. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  25. 2026SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinementcited by 24
  26. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 23
  27. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  28. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  29. 2025UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoningcited by 21
  30. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  31. 2025Towards Automating Data Access Permissions in AI Agentscited by 20
  32. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systemscited by 20
  33. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  34. 2024Defining and Detecting the Defects of the Large Language Model-based Autonomous Agentscited by 14
  35. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  36. 2025AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defensescited by 14
  37. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 14
  38. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Assessing Automated Prompt Injection Attacks in Agentic Environments
  2. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks
  3. 2026AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agents
  4. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis
  5. 2025Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?
  6. 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
  7. 2026AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents
  8. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks