mlsanityData to 7 Oct 2026

AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways

arXiv 2406.02630 ↗First posted 4 Jun 2024cs.CR cs.AIcited by 122

Not on the map. Why?

Cited by 90 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. AI agents, capable of perceiving user inputs, reasoning and planning tasks, and executing actions, have seen remarkable advancements in algorithm development and task performance. However, the security challenges they pose remain under-explored and unresolved. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps: unpredictability of multi-step user inputs, complexity in internal executions, variability of operational environments, and interactions with untrusted external entities. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents. The insights provided aim to inspire further research into addressing the security threats associated with AI agents, thereby fostering the development of more robust and secure AI agent applications.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Builds on 152 of its 243 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

152 of 243 references.

References · 152

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  13. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  14. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  15. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  16. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  17. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  18. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  19. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  20. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  21. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  22. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  23. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  24. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  25. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  26. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  27. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  28. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  29. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  30. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  31. 2023Detecting Language Model Attacks with Perplexitycited by 411
  32. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  33. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  34. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  35. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  36. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  37. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  38. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347

50 most cited of 152 All 152 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 48; 2026: 68.

122 papers.

Sort:

By area:

  1. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 257
  2. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  3. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  4. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  5. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  6. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  8. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  9. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 58
  10. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  11. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  12. 2025GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modelingcited by 35
38 more
  1. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  2. 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 34
  3. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 31
  4. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  5. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  6. 2024Security of AI Agentscited by 24
  7. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  8. 2025MCPXKIT: The Unified Toolkit for Analyzing Model Context Protocol Securitycited by 21
  9. 2026A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypescited by 19
  10. 2026Quantifying Frontier LLM Capabilities for Container Sandbox Escapecited by 18
  11. 2026ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watcherscited by 18
  12. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  13. 2025Beyond the Protocol: Unveiling Attack Vectors in the Model Context Protocol (MCP) Ecosystemcited by 17
  14. 2026Runtime Governance for AI Agents: Policies on Pathscited by 17
  15. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 17
  16. 2024A Study on Prompt Injection Attack Against LLM-Integrated Mobile Robotic Systemscited by 15
  17. 2024Defining and Detecting the Defects of the Large Language Model-based Autonomous Agentscited by 14
  18. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 14
  19. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  20. 2025A Survey on Agent Workflow -- Status and Futurecited by 14
  21. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomycited by 14
  22. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  23. 2025From Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systemscited by 12
  24. 2025The Hidden Dangers of Browsing AI Agentscited by 11
  25. 2025CyberRAG: An Agentic RAG cyber attack classification and reporting toolcited by 11
  26. 2025The Trust Paradox in LLM-Based Multi-Agent Systems: When Collaboration Becomes a Security Vulnerabilitycited by 11
  27. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  28. 2025Agent Safety Alignment via Reinforcement Learningcited by 10
  29. 2026Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystemcited by 10
  30. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  31. 2026AgentGuardian: Learning Access Control Policies to Govern AI Agent Behaviorcited by 9
  32. 2025LLM Applications: Current Paradigms and the Next Frontiercited by 8
  33. 2025Agent Guide: A Simple Agent Behavioral Watermarking Frameworkcited by 8
  34. 2026LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarioscited by 8
  35. 2025Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsingcited by 7
  36. 2025Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigmscited by 7
  37. 2025BetaWeb: Towards a Blockchain-enabled Trustworthy Agentic Webcited by 7
  38. 2025Takedown: How It's Done in Modern Coding Agent Exploitscited by 6

50 most cited of 122 All 122 (JSON)