mlsanityData to 14 Oct 2026

Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents

arXiv 2503.00061 ↗First posted 27 Feb 2025cs.CR cs.LGcited by 125

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Large Language Model (LLM) agents exhibit remarkable performance across diverse applications by using external tools to interact with environments. However, integrating external tools introduces security risks, such as indirect prompt injection (IPI) attacks. Despite defenses designed for IPI attacks, their robustness remains questionable due to insufficient testing against adaptive attacks. In this paper, we evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%. This reveals critical vulnerabilities in current defenses. Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability. The code is available at https://github.com/uiuc-kang-lab/AdaptiveAttackAgent.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 41 of its 58 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

41 of 58 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  2. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
  3. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul
  4. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  5. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  6. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
  7. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  8. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  9. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  10. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  11. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  12. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  10. 2023Large Language Modelscited by 780
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  12. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
17 more
  1. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  2. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 546
  3. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  4. 2023Detecting Language Model Attacks with Perplexitycited by 431
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  6. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  7. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  8. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 170
  9. 2023A Language Agent for Autonomous Drivingcited by 164
  10. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 164
  11. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  12. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 106
  13. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
  14. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  15. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56
  16. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  17. 2021LAFEAT: Piercing Through Adversarial Defenses with Latent Featurescited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 33; 2026: 92.

125 papers.

Sort:

By area:

  1. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  2. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  3. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 87
  4. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 58
  5. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agentscited by 54
  6. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 54
  7. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 38
  8. 2025Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agentscited by 36
  9. 2025Agent Skills Enable a New Class of Realistic and Trivially Simple Prompt Injectionscited by 33
  10. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agentscited by 31
  11. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 30
  12. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
38 more
  1. 2025Free-MAD: Consensus-Free Multi-Agent Debatecited by 29
  2. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purificationcited by 29
  3. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  4. 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competitioncited by 24
  5. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  6. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 20
  7. 2026Prompt Injection as Role Confusioncited by 17
  8. 2025Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instructioncited by 16
  9. 2025FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agentscited by 16
  10. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  11. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 15
  12. 2026Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agentscited by 15
  13. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injectioncited by 14
  14. 2025Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Treecited by 14
  15. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agentscited by 14
  16. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 14
  17. 2026AdapTools: Adaptive Tool-based Indirect Prompt Injection Attacks on Agentic LLMscited by 13
  18. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 12
  19. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  20. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  21. 2026Optimizing Agent Planning for Security and Autonomycited by 10
  22. 2026What Makes a Good LLM Agent for Real-world Penetration Testing?cited by 10
  23. 2026ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correctioncited by 10
  24. 2026Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisionscited by 10
  25. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 9
  26. 2026Defense Against Indirect Prompt Injection via Tool Result Parsingcited by 9
  27. 2026Reframing LLM Agent Security as an Agent-Human Interaction Problemcited by 9
  28. 2025STAC: When Innocent Tools Form Dangerous Chains for LLM Agentscited by 8
  29. 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agentscited by 8
  30. 2025VortexPIA: Indirect Prompt Injection Attack against LLMs for Efficient Extraction of User Privacycited by 7
  31. 2026The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societiescited by 7
  32. 2026Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skillscited by 7
  33. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 7
  34. 2026Automating Agent Hijacking via Structural Template Injectioncited by 6
  35. 2025Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessmentcited by 5
  36. 2026ER-MIA: Black-Box Adversarial Memory Injection Attacks on Long-Term Memory-Augmented Large Language Modelscited by 5
  37. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  38. 2026AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agentscited by 5

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Your Agent is More Brittle Than You Think: Uncovering Indirect Injection Vulnerabilities in Agentic LLMs
  2. 2026Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
  3. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  4. 2025A Critical Evaluation of Defenses against Prompt Injection Attacks
  5. 2026Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents
  6. 2025Design Patterns for Securing LLM Agents against Prompt Injections
  7. 2024Systematically Analyzing Prompt Injection Vulnerabilities in Diverse LLM Architectures
  8. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Survey