mlsanityData to 14 Oct 2026

The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies

arXiv 2407.19354 ↗First posted 28 Jul 2024cs.CRcited by 131

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Inspired by the rapid development of Large Language Models (LLMs), LLM agents have evolved to perform complex tasks. LLM agents are now extensively applied across various domains, handling vast amounts of data to interact with humans and execute tasks. The widespread applications of LLM agents demonstrate their significant commercial value; however, they also expose security and privacy vulnerabilities. At the current stage, comprehensive research on the security and privacy of LLM agents is highly needed. This survey aims to provide a comprehensive overview of the newly emerged privacy and security issues faced by LLM agents. We begin by introducing the fundamental knowledge of LLM agents, followed by a categorization and analysis of the threats. We then discuss the impacts of these threats on humans, environment, and other agents. Subsequently, we review existing defensive strategies, and finally explore future trends. Additionally, the survey incorporates diverse case studies to facilitate a more accessible understanding. By highlighting these critical security and privacy issues, the survey seeks to stimulate future research towards enhancing the security and privacy of LLM agents, thereby increasing their reliability and trustworthiness in future applications.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 84 of its 143 references that are in the map. First posted on arXiv 28 Jul 2024.

Builds on

84 of 143 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities10 Jul
  2. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agents8 May
  3. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilities11 Apr
  4. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  5. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  6. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  7. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
  8. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents20 Sep

Other references · 76

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  7. 2023A Survey on Multimodal Large Language Modelscited by 909
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  12. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
38 more
  1. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  2. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  4. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  5. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  6. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  8. 2024Understanding the planning of LLM agents: A surveycited by 421
  9. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  10. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  11. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  12. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  13. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  14. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  15. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  16. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  17. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  18. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  19. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  20. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  21. 2023Poisoning Language Models During Instruction Tuningcited by 269
  22. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  23. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  24. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  25. 2024Large Language Models for Education: A Survey and Outlookcited by 188
  26. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  27. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  28. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 170
  29. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 165
  30. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  31. 2023User Behavior Simulation with Large Language Model based Agentscited by 140
  32. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 131
  33. 2020Data-Free Model Extractioncited by 124
  34. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 124
  35. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  36. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
  37. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
  38. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 54; 2026: 71.

131 papers.

Sort:

By area:

  1. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  2. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  3. 2025Progent: Securing AI Agents with Privilege Controlcited by 141
  4. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 125
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  6. 2024GUI Agents: A Surveycited by 103
  7. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 101
  8. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  9. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  10. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 62
  11. 2025LLM Agents for Education: Advances and Applicationscited by 59
  12. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
38 more
  1. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  2. 2025Measuring Agents in Productioncited by 37
  3. 2025Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agentscited by 32
  4. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
  5. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  6. 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 27
  7. 2024When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?cited by 24
  8. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 24
  9. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  10. 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agentcited by 23
  11. 2025SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraintscited by 23
  12. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  13. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  14. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 22
  15. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  16. 2026A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypescited by 22
  17. 2025CORBA: Contagious Recursive Blocking Attacks on Multi-Agent Systems Based on Large Language Modelscited by 20
  18. 2026Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montagecited by 19
  19. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  20. 2025From Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systemscited by 14
  21. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correctioncited by 14
  22. 2025A Survey on Agent Workflow -- Status and Futurecited by 14
  23. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  24. 2025EgoNormia: Benchmarking Physical Social Norm Understandingcited by 13
  25. 2026Towards Secure Agent Skills: Architecture, Threat Taxonomy, and Security Analysiscited by 13
  26. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 12
  27. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 12
  28. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  29. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 11
  30. 2024Understanding Large Language Models in Your Pockets: Performance Study on COTS Mobile Devicescited by 10
  31. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 10
  32. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  33. 2025Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactionscited by 10
  34. 2025LLM Applications: Current Paradigms and the Next Frontiercited by 8
  35. 2025Position: Collaborative Agentic AI Needs Interoperability Across Ecosystemscited by 8
  36. 2026AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networkscited by 7
  37. 2026Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agentscited by 7
  38. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents
  2. 2025SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
  3. 2025Security Concerns for Large Language Models: A Survey
  4. 2024When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs
  5. 2024Security and Privacy Challenges of Large Language Models: A Survey
  6. 2025A Survey of Attacks on Large Language Models
  7. 2025LLM Agents Should Employ Security Principles
  8. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents