mlsanityData to 7 Oct 2026

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

arXiv 2403.02691 ↗First posted 5 Mar 2024cs.CL cs.CRcited by 476

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Recent work has embodied LLMs as agents, allowing them to access tools, perform actions, and interact with external content (e.g., emails or websites). However, external content introduces the risk of indirect prompt injection (IPI) attacks, where malicious instructions are embedded within the content processed by LLMs, aiming to manipulate these agents into executing detrimental actions against users. Given the potentially severe consequences of such attacks, establishing benchmarks to assess and mitigate these risks is imperative. In this work, we introduce InjecAgent, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks. InjecAgent comprises 1,054 test cases covering 17 different user tools and 62 attacker tools. We categorize attack intentions into two primary types: direct harm to users and exfiltration of private data. We evaluate 30 different LLM agents and show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time. Further investigation into an enhanced setting, where the attacker instructions are reinforced with a hacking prompt, shows additional increases in success rates, nearly doubling the attack success rate on the ReAct-prompted GPT-4. Our findings raise questions about the widespread deployment of LLM Agents. Our benchmark is available at https://github.com/uiuc-kang-lab/InjecAgent.

Key facts. Cited by 476 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 30 of its 94 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

30 of 94 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov
  2. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
16 more
  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  6. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  8. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  9. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  10. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  11. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  12. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  13. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  14. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmarkcited by 54
  15. 2023Assessing Prompt Injection Risks in 200+ Custom GPTscited by 49
  16. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 116; 2026: 326.

476 papers.

Sort:

By area:

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  3. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  4. 2024Tool Learning with Large Language Models: A Surveycited by 221
  5. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  6. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  7. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  8. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  9. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  10. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  11. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
38 more
  1. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  2. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systemscited by 118
  3. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  4. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2025Securing AI Agents with Information-Flow Controlcited by 108
  7. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  8. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  9. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  10. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  11. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  12. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 90
  13. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 89
  14. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  15. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  16. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  17. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  18. 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakagecited by 64
  19. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  20. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  21. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 59
  22. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 57
  23. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 56
  24. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  25. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 52
  26. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 49
  27. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
  28. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agentscited by 46
  29. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  30. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agentscited by 42
  31. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  32. 2025BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attackscited by 37
  33. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 37
  34. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  35. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  36. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  37. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 32
  38. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 31

50 most cited of 476 All 476 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Prompt Injection to Web Exploitation: Revisiting Classic Vulnerabilities in LLM-Integrated Applications
  2. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks
  3. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltration
  4. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  5. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  6. 2026NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations
  7. 2026ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents
  8. 2025Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs