mlsanityData to 14 Oct 2026

The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey

arXiv 2603.11088 ↗First posted 11 Mar 2026cs.CR cs.AIcited by 28

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

AI agents that combine large language models with non-AI system components are rapidly emerging in real-world applications, offering unprecedented automation and flexibility. However, this unprecedented flexibility introduces complex security challenges fundamentally different from those in traditional software systems. This paper presents the first systematic and comprehensive survey of AI agent security, including an analysis of the design space, attack landscape, and defense mechanisms for secure AI agent systems. We further conduct case studies to point out existing gaps in securing agentic AI systems and identify open challenges in this emerging domain. Our work also introduces the first systematic framework for understanding the security risks and defense strategies of AI agents, serving as a foundation for building both secure agentic systems and advancing research in this critical area.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 75 of its 150 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

75 of 150 references.

Closest ancestors · 30

Same direction, newest first

  1. 2025Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents22 Sep
  2. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents21 Aug
  3. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection2 Aug
  4. 2025PromptArmor: Simple yet Effective Prompt Injection Defenses21 Jul
  5. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents13 Jun
  6. 2025Design Patterns for Securing LLM Agents against Prompt Injections10 Jun
  7. 2025SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems9 Jun
  8. 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents31 May
  9. 2025LLM Agents Should Employ Security Principles29 May
  10. 2025GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling25 May
  11. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents9 May
  12. 2025LlamaFirewall: An open source guardrail system for building secure AI agents6 May
18 more
  1. 2025Prompt Injection Attack to Tool Selection in LLM Agents28 Apr
  2. 2025SAGA: A Security Architecture for Governing AI Agentic Systems27 Apr
  3. 2025Progent: Securing AI Agents with Privilege Control16 Apr
  4. 2025MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits2 Apr
  5. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
  6. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents17 Mar
  7. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection17 Feb
  8. 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage13 Feb
  9. 2025Firewalls to Secure Dynamic LLM Agentic Networks3 Feb
  10. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents21 Dec
  11. 2024Imprompter: Tricking LLM Agents into Improper Tool Use19 Oct
  12. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  13. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
  14. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  15. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
  16. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agents8 May
  17. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  18. 2023Testing Language Model Agents Safely in the Wild17 Nov

Other references · 45

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  9. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  10. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  11. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  12. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
33 more
  1. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  2. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  3. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  4. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  5. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  6. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  7. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  8. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 159
  9. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  10. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  11. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  12. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 96
  13. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 81
  14. 2024Visibility into AI Agentscited by 79
  15. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 67
  16. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
  17. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  18. 2025Contextual Agent Security: A Policy for Every Purposecited by 59
  19. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 55
  20. 2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 51
  21. 2025PromptShield: Deployable Detection for Prompt Injection Attackscited by 50
  22. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 42
  23. 2024Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agentscited by 40
  24. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 36
  25. 2025MCIP: Protecting MCP Safety via Model Contextual Integrity Protocolcited by 34
  26. 2024Adaptive Deployment of Untrusted LLMs Reduces Distributed Threatscited by 31
  27. 2025Agent Network Protocol Technical White Papercited by 25
  28. 2025Towards Automating Data Access Permissions in AI Agentscited by 21
  29. 2023Towards more Practical Threat Models in Artificial Intelligence Securitycited by 17
  30. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  31. 2015Android Permissions Remystified: A Field Study on Contextual Integritycited by 15
  32. 2025Position: Contextual Integrity is Inadequately Applied to Language Modelscited by 11
  33. 2025HONEYBEE: Efficient Role-based Access Control for Vector Databases via Dynamic Partitioning[Technical Report]cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 28.

28 papers.

Sort:

By area:

  1. 2026ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watcherscited by 20
  2. 2026MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skillscited by 19
  3. 2026Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agentscited by 13
  4. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 11
  5. 2026VIGIL: Runtime Enforcement of Behavioral Specifications in AI Agent Skillscited by 9
  6. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 8
  7. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 8
  8. 2026AI Agents Under EU Lawcited by 5
  9. 2026Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lenscited by 5
  10. 2026EVOMAL: Self-Poisoning in Self-Evolving Coding Agentscited by 3
  11. 2026SUDP: Secret-Use Delegation Protocol for Agentic Systemscited by 2
  12. 2026Protocol-Level Attacks on Agentic Commerce Platforms: A Cross-Platform Taxonomy, AIP-Bench, and Unified Defensecited by 2
16 more
  1. 2026AgentFlow: A Flow-Centric Policy Language and Framework for Securing LLM Agent Systemscited by 2
  2. 2026Content-Aware Attack Detection in LLM Agent Tool-Call Traffic: An Empirical Study of Features, Architectures, and Evaluation Protocolscited by 1
  3. 2026Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attackscited by 1
  4. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  5. 2026Rethinking Penetration Testing for AI-Enabled Systems: From Resource Compromise to Behavioral Objective Violationcited by 0
  6. 2026Where Is the Tradeoff in Using Third-Party API Routers for Agentic Software Development?cited by 0
  7. 2026Formal Verification of Agentic Systems over Operational Datacited by 0
  8. 2026Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Agentscited by 0
  9. 2026RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguardscited by 0
  10. 2026Value-Preserving Architectures for Agentic AI Systemscited by 0
  11. 2026A-JIT: Agentic Just-In-Time Software Constructioncited by 0
  12. 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systemscited by 0
  13. 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challengescited by 0
  14. 2026Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluationcited by 0
  15. 2026ChronosAttack: Adversarial Tool Scheduling Attacks on LLM Agentscited by 0
  16. 2026VEPA: Verified Evolving Formal Policy for Lifelong LLM Agent Guardrailscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
  2. 2024Security of AI Agents
  3. 2025Systems Security Foundations for Agentic Computing
  4. 2026Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges
  5. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomy
  6. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
  7. 2026A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Development
  8. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks