mlsanityData to 7 Oct 2026

Security of AI Agents

arXiv 2406.08689 ↗First posted 12 Jun 2024cs.CR cs.AIcited by 24

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

AI agents have been boosted by large language models. AI agents can function as intelligent assistants and complete tasks on behalf of their users with access to tools and the ability to execute commands in their environments. Through studying and experiencing the workflow of typical AI agents, we have raised several concerns regarding their security. These potential vulnerabilities are not addressed by the frameworks used to build the agents, nor by research aimed at improving the agents. In this paper, we identify and describe these vulnerabilities in detail from a system security perspective, emphasizing their causes and severe effects. Furthermore, we introduce defense mechanisms corresponding to each vulnerability with design and experiments to evaluate their viability. Altogether, this paper contextualizes the security issues in the current development of AI agents and delineates methods to make AI agents safer and more reliable.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 70 of its 122 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

70 of 122 references.

References · 70

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  13. 2022Red Teaming Language Models with Language Modelscited by 983
  14. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  15. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  16. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  17. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  18. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  19. 2023Cognitive Architectures for Language Agentscited by 422
  20. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  21. 2024Understanding the planning of LLM agents: A surveycited by 414
  22. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  23. 2023Language Models can Solve Computer Taskscited by 371
  24. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  25. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  26. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  27. 2017Certified Defenses for Data Poisoning Attackscited by 288
  28. 2023Language Modeling Is Compressioncited by 225
  29. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  30. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  31. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  32. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  33. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  34. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  35. 2017A Survey on Homomorphic Encryption Schemes: Theory and Implementationcited by 154
  36. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  37. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  38. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 9.

24 papers.

Sort:

By area:

  1. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  2. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  3. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  4. 2025Fully Autonomous AI Agents Should Not be Developedcited by 43
  5. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  6. 2025A Survey on Agent Workflow -- Status and Futurecited by 14
  7. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12
  8. 2024Security Threats in Agentic AI Systemcited by 11
  9. 2025The Hidden Dangers of Browsing AI Agentscited by 11
  10. 2026OpenClaw PRISM: A Zero-Fork, Defense-in-Depth Runtime Security Layer for Tool-Augmented LLM Agentscited by 10
  11. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 7
  12. 2025Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impactcited by 6
12 more
  1. 2026Autonomous Action Runtime Management(AARM):A System Specification for Securing AI-Driven Actions at Runtimecited by 5
  2. 2025Penetration Testing of Agentic AI: A Comparative Security Analysis Across Models and Frameworkscited by 4
  3. 2025AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretabilitycited by 3
  4. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltrationcited by 3
  5. 2026Toward Securing AI Agents Like Operating Systemscited by 2
  6. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  7. 2026Anonymous-by-Construction: An LLM-Driven Framework for Privacy-Preserving Textcited by 1
  8. 2026ContractBench: Can LLM Agents Preserve Observation Contracts?cited by 1
  9. 2025Generative AI in Heritage Practice: Improving the Accessibility of Heritage Guidancecited by 0
  10. 2026Human Attribution of Causality to AI Across Agency, Misuse, and Misalignmentcited by 0
  11. 2026A Policy Algebra for Trust-Preserving Agentic AI Executioncited by 0
  12. 2026Runtime Authorization Consistency Checking for MCP-based Agentic Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
  2. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
  3. 2025Systems Security Foundations for Agentic Computing
  4. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks
  5. 2024Safeguarding AI Agents: Developing and Analyzing Safety Architectures
  6. 2026Rethinking Agent Security as a Networking Problem
  7. 2026Security Considerations for Artificial Intelligence Agents
  8. 2026AgentWard: A Lifecycle Security Architecture for Autonomous AI Agents