SoK: The Attack Surface of Agentic AI - Tools and Autonomy
On the map: Retrieval-augmented generation → Privacy and security of RAG
also in 2 more
Abstract (arXiv)
Recent AI systems combine large language models with tools, external knowledge via retrieval-augmented generation (RAG), and even autonomous multi-agent decision loops. This agentic AI paradigm greatly expands capabilities - but also vastly enlarges the attack surface. In this systematization, we map out the trust boundaries and security risks of agentic LLM-based systems. We develop a comprehensive taxonomy of attacks spanning prompt-level injections, knowledge-base poisoning, tool/plug-in exploits, and multi-agent emergent threats. Through a detailed literature review, we synthesize evidence from 2023-2025, including more than 20 peer-reviewed and archival studies, industry reports, and standards. We find that agentic systems introduce new vectors for indirect prompt injection, code execution exploits, RAG index poisoning, and cross-agent manipulation that go beyond traditional AI threats. We define attacker models and threat scenarios, and propose metrics (e.g., Unsafe Action Rate, Privilege Escalation Distance) to evaluate security posture. Our survey examines defenses such as input sanitization, retrieval filters, sandboxes, access control, and "AI guardrails," assessing their effectiveness and pointing out the areas where protection is still lacking. To assist practitioners, we outline defensive controls and provide a phased security checklist for deploying agentic AI (covering design-time hardening, runtime monitoring, and incident response). Finally, we outline open research challenges in secure autonomous AI (robust tool APIs, verifiable agent behavior, supply-chain safeguards) and discuss ethical and responsible disclosure practices. We systematize recent findings to help researchers and engineers understand and mitigate security risks in agentic AI.
Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 16 of its 30 references that are in the map. First posted on arXiv 24 Mar 2026.
Builds on
16 of 30 references.
Closest ancestors · 1
Same direction, newest first
Other references · 15
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
- 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
- 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
- 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
- 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
- 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
- 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
- 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
- 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
- 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
3 more
- 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
- 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
- 2025Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agentscited by 32
Built on by
Citing papers per year: 2026: 14.
14 papers.
Sort:
By area:
- 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 6
- 2026A Formal Security Framework for MCP-Based AI Agents: Threat Taxonomy, Verification Models, and Defense Mechanismscited by 2
- 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environmentscited by 2
- 2026Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedbackcited by 2
- 2026A Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problemscited by 0
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilitiescited by 0
- 2026LLM-Centric Agentic AI for UAV Swarms: Architecture, Enabling Technologies, and Open Problemscited by 0
- 2026From Agent Failure Paths to Quantified Residual Risk: A Compositional Framework for Resilient Agentic AIcited by 0
- 2026Where Is the Tradeoff in Using Third-Party API Routers for Agentic Software Development?cited by 0
- 2026Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failurescited by 0
- 2026Trust propagation and structural containment in Multi-agent LLM pipelinescited by 0
- 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systemscited by 0
- 2026Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation21 Sep 2026
- 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges20 Sep 2026
- 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systems20 Sep 2026
- 2026Trust propagation and structural containment in Multi-agent LLM pipelines15 Sep 2026
- 2026Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failures14 Sep 2026
- 2026Where Is the Tradeoff in Using Third-Party API Routers for Agentic Software Development?26 Jul 2026
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities7 Jul 2026
- 2026LLM-Centric Agentic AI for UAV Swarms: Architecture, Enabling Technologies, and Open Problems5 Jul 2026
- 2026A Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problems30 Jun 2026
- 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies22 Jun 2026
- 2026Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback17 May 2026
- 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments11 May 2026
- 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 6
- 2026A Formal Security Framework for MCP-Based AI Agents: Threat Taxonomy, Verification Models, and Defense Mechanismscited by 2
- 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environmentscited by 2
- 2026Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedbackcited by 2
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilitiescited by 0
- 2026LLM-Centric Agentic AI for UAV Swarms: Architecture, Enabling Technologies, and Open Problemscited by 0
- 2026From Agent Failure Paths to Quantified Residual Risk: A Compositional Framework for Resilient Agentic AIcited by 0
- 2026Where Is the Tradeoff in Using Third-Party API Routers for Agentic Software Development?cited by 0
- 2026Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failurescited by 0
- 2026Trust propagation and structural containment in Multi-agent LLM pipelinescited by 0
- 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systemscited by 0
- 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challengescited by 0
12 most cited of 13 All 13 (JSON)
- 2026Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation21 Sep 2026
- 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges20 Sep 2026
- 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systems20 Sep 2026
- 2026Trust propagation and structural containment in Multi-agent LLM pipelines15 Sep 2026
- 2026Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failures14 Sep 2026
- 2026Where Is the Tradeoff in Using Third-Party API Routers for Agentic Software Development?26 Jul 2026
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities7 Jul 2026
- 2026LLM-Centric Agentic AI for UAV Swarms: Architecture, Enabling Technologies, and Open Problems5 Jul 2026
- 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies22 Jun 2026
- 2026Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback17 May 2026
- 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments11 May 2026
- 2026From Agent Failure Paths to Quantified Residual Risk: A Compositional Framework for Resilient Agentic AI28 Apr 2026
12 newest of 13 All 13 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework
- 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
- 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
- 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks
- 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
- 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
- 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
- 2025Systems Security Foundations for Agentic Computing