# Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges

arXiv: https://arxiv.org/abs/2510.23883 · HTML: https://mlsanity.com/paper/2510.23883/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 192 of its 288 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 30 of 192)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)

## Built on by (most cited first, 28 of 28)

- [Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems](https://mlsanity.com/paper/2601.17548/) (2026)
- [A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes](https://mlsanity.com/paper/2601.05293/) (2026)
- [Systematization of Knowledge: Security and Safety in the Model Context Protocol Ecosystem](https://mlsanity.com/paper/2512.08290/) (2025)
- [Towards Secure Agent Skills: Architecture, Threat Taxonomy, and Security Analysis](https://mlsanity.com/paper/2604.02837/) (2026)
- [A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework](https://mlsanity.com/paper/2604.23338/) (2026)
- [Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode](https://mlsanity.com/paper/2604.04978/) (2026)
- [Agentic Business Process Management: A Research Manifesto](https://mlsanity.com/paper/2603.18916/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations](https://mlsanity.com/paper/2512.22883/) (2025)
- [Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents](https://mlsanity.com/paper/2605.16282/) (2026)
- [Enforcing Benign Trajectories: A Behavioral Firewall for Structured-Workflow AI Agents](https://mlsanity.com/paper/2604.26274/) (2026)
- [From Secure Agentic AI to Secure Agentic Web: Challenges, Threats, and Future Directions](https://mlsanity.com/paper/2603.01564/) (2026)
- [Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response](https://mlsanity.com/paper/2607.25379/) (2026)
- [Securing Agentic AI: From Per-Action Checks to Trajectory Assurance](https://mlsanity.com/paper/2608.01558/) (2026)
- [S3: Improving Agent Safety through Multi-Stage Defense](https://mlsanity.com/paper/2608.02683/) (2026)
- [Blockchain Empowered Trustworthy Agent Networks: Foundations, Taxonomy, and Future Directions](https://mlsanity.com/paper/2608.04626/) (2026)
- [Intellicise Wireless Networks Meet Agentic AI: A Security and Privacy Perspective](https://mlsanity.com/paper/2602.15290/) (2026)
- [Don't Vibe Code, Do Skele-Code: Interactive No-Code Notebooks for Subject Matter Experts to Build Lower-Cost Agentic Workflows](https://mlsanity.com/paper/2603.18122/) (2026)
- [Can a Single Message Paralyze the AI Infrastructure? The Rise of AbO-DDoS Attacks through Targeted Mobius Injection](https://mlsanity.com/paper/2605.11442/) (2026)
- [Surviving the Unseen: Predictive Defense for Novel Multi-Turn Multimodal Attacks](https://mlsanity.com/paper/2605.18988/) (2026)
- [Agyn: An Open-Source Platform for AI Agents with Scalable On-Demand Execution, Agent Definition as a Code, and Zero-Trust Access](https://mlsanity.com/paper/2605.27575/) (2026)
- [A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Development](https://mlsanity.com/paper/2606.14816/) (2026)
- [LLM-Centric Agentic AI for UAV Swarms: Architecture, Enabling Technologies, and Open Problems](https://mlsanity.com/paper/2607.09756/) (2026)
- [Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability](https://mlsanity.com/paper/2607.10402/) (2026)
- [On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models](https://mlsanity.com/paper/2608.10530/) (2026)
- [Emergence of Agentic AI: A Review on Evolution, Background, Working Principles, Applications, Adoption Factors, and Future Research Directions](https://mlsanity.com/paper/2608.18110/) (2026)
- [Unified Agentic Video Editing Across Levels of Complexity and Creativity](https://mlsanity.com/paper/2609.12769/) (2026)
- [Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges](https://mlsanity.com/paper/2609.23894/) (2026)

## Abstract (arXiv)

Agentic AI systems powered by large language models (LLMs) and endowed with planning, tool use, memory, and autonomy, are emerging as powerful, flexible platforms for automation. Their ability to autonomously execute tasks across web, software, and physical environments creates new and amplified security risks, distinct from both traditional AI safety and conventional software security. This survey outlines a taxonomy of threats specific to agentic AI, reviews recent benchmarks and evaluation methodologies, and discusses defense strategies from both technical and governance perspectives. We synthesize current research and highlight open challenges, aiming to support the development of secure-by-design agent systems.