mlsanityData to 7 Oct 2026

Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges

arXiv 2510.23883 ↗First posted 27 Oct 2025cs.AIcited by 26

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Agentic AI systems powered by large language models (LLMs) and endowed with planning, tool use, memory, and autonomy, are emerging as powerful, flexible platforms for automation. Their ability to autonomously execute tasks across web, software, and physical environments creates new and amplified security risks, distinct from both traditional AI safety and conventional software security. This survey outlines a taxonomy of threats specific to agentic AI, reviews recent benchmarks and evaluation methodologies, and discusses defense strategies from both technical and governance perspectives. We synthesize current research and highlight open challenges, aiming to support the development of secure-by-design agent systems.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 192 of its 288 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

192 of 288 references.

Closest ancestors · 50

Same direction, newest first

  1. 2025Agentic Misalignment: How LLMs Could Be Insider Threats5 Oct
  2. 2025MCPSecBench: A Systematic Security Benchmark and Playground for Testing Model Context Protocols17 Aug
  3. 2025PromptArmor: Simple yet Effective Prompt Injection Defenses21 Jul
  4. 2025Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree20 Jul
  5. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety8 Jul
  6. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows29 Jun
  7. 2025Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation26 Jun
  8. 2025MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation25 Jun
  9. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents17 Jun
  10. 2025Design Patterns for Securing LLM Agents against Prompt Injections10 Jun
  11. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems4 Jun
  12. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systems28 May
38 more
  1. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks22 Apr
  2. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injection19 Apr
  3. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threats18 Apr
  4. 2025textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks31 Mar
  5. 2025DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agents31 Mar
  6. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
  7. 2025Defeating Prompt Injections by Design24 Mar
  8. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents24 Mar
  9. 2025SafeArena: Evaluating the Safety of Autonomous Web Agents6 Mar
  10. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents27 Feb
  11. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attacks20 Feb
  12. 2025Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks18 Feb
  13. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks12 Feb
  14. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents7 Feb
  15. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents21 Dec
  16. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  17. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents17 Dec
  18. 2024Attacking Vision-Language Computer Agents via Pop-ups4 Nov
  19. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agents22 Oct
  20. 2024Imprompter: Tricking LLM Agents into Improper Tool Use19 Oct
  21. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  22. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems9 Oct
  23. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents9 Oct
  24. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  25. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
  26. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
  27. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul
  28. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  29. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  30. 2024Dissecting Adversarial Robustness of Multimodal LM Agents18 Jun
  31. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilities11 Apr
  32. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems8 Mar
  33. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  34. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganography12 Feb
  35. 2024LLM Agents can Autonomously Hack Websites6 Feb
  36. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  37. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  38. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 142

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  12. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
38 more
  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2025s1: Simple test-time scalingcited by 1,311
  8. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  13. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  14. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  15. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  16. 2023A Comprehensive Overview of Large Language Modelscited by 639
  17. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  18. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  19. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  20. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  21. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  22. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  23. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  24. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  25. 2023Detecting Language Model Attacks with Perplexitycited by 411
  26. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  27. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  28. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  29. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  30. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  31. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 281
  32. 2019Fooling automated surveillance cameras: adversarial patches to attack person detectioncited by 268
  33. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 266
  34. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  35. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  36. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  37. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  38. 2024AI Agents That Mattercited by 199

50 most cited of 142 All 142 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 24.

26 papers.

Sort:

By area:

  1. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  2. 2026A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypescited by 19
  3. 2025Systematization of Knowledge: Security and Safety in the Model Context Protocol Ecosystemcited by 14
  4. 2026Towards Secure Agent Skills: Architecture, Threat Taxonomy, and Security Analysiscited by 11
  5. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 9
  6. 2026Agentic Business Process Management: A Research Manifestocited by 7
  7. 2025Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundationscited by 4
  8. 2026Enforcing Benign Trajectories: A Behavioral Firewall for Structured-Workflow AI Agentscited by 3
  9. 2026From Secure Agentic AI to Secure Agentic Web: Challenges, Threats, and Future Directionscited by 1
  10. 2026Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agentscited by 1
  11. 2026Securing Agentic AI: From Per-Action Checks to Trajectory Assurancecited by 1
  12. 2026S3: Improving Agent Safety through Multi-Stage Defensecited by 1
14 more
  1. 2026Blockchain Empowered Trustworthy Agent Networks: Foundations, Taxonomy, and Future Directionscited by 1
  2. 2026Intellicise Wireless Networks Meet Agentic AI: A Security and Privacy Perspectivecited by 0
  3. 2026Don't Vibe Code, Do Skele-Code: Interactive No-Code Notebooks for Subject Matter Experts to Build Lower-Cost Agentic Workflowscited by 0
  4. 2026Can a Single Message Paralyze the AI Infrastructure? The Rise of AbO-DDoS Attacks through Targeted Mobius Injectioncited by 0
  5. 2026Surviving the Unseen: Predictive Defense for Novel Multi-Turn Multimodal Attackscited by 0
  6. 2026Agyn: An Open-Source Platform for AI Agents with Scalable On-Demand Execution, Agent Definition as a Code, and Zero-Trust Accesscited by 0
  7. 2026A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Developmentcited by 0
  8. 2026LLM-Centric Agentic AI for UAV Swarms: Architecture, Enabling Technologies, and Open Problemscited by 0
  9. 2026Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerabilitycited by 0
  10. 2026Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Responsecited by 0
  11. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Modelscited by 0
  12. 2026Emergence of Agentic AI: A Review on Evolution, Background, Working Principles, Applications, Adoption Factors, and Future Research Directionscited by 0
  13. 2026Unified Agentic Video Editing Across Levels of Complexity and Creativitycited by 0
  14. 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challengescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
  2. 2025Systems Security Foundations for Agentic Computing
  3. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomy
  4. 2025A Survey on Agentic Security: Applications, Threats and Defenses
  5. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
  6. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
  7. 2025Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agents
  8. 2025A Safety and Security Framework for Real-World Agentic Systems