mlsanityData to 7 Oct 2026

A Survey on Trustworthy LLM Agents: Threats and Countermeasures

arXiv 2503.09648 ↗First posted 12 Mar 2025cs.MA cs.CYcited by 93

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

With the rapid evolution of Large Language Models (LLMs), LLM-based agents and Multi-agent Systems (MAS) have significantly expanded the capabilities of LLM ecosystems. This evolution stems from empowering LLMs with additional modules such as memory, tools, environment, and even other agents. However, this advancement has also introduced more complex issues of trustworthiness, which previous research focused solely on LLMs could not cover. In this survey, we propose the TrustAgent framework, a comprehensive study on the trustworthiness of agents, characterized by modular taxonomy, multi-dimensional connotations, and technical implementation. By thoroughly investigating and summarizing newly emerged attacks, defenses, and evaluation methods for agents and MAS, we extend the concept of Trustworthy LLM to the emerging paradigm of Trustworthy Agent. In TrustAgent, we begin by deconstructing and introducing various components of the Agent and MAS. Then, we categorize their trustworthiness into intrinsic (brain, memory, and tool) and extrinsic (user, agent, and environment) aspects. Subsequently, we delineate the multifaceted meanings of trustworthiness and elaborate on the implementation techniques of existing research related to these internal and external modules. Finally, we present our insights and outlook on this domain, aiming to provide guidance for future endeavors.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 84 of its 148 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

84 of 148 references.

Closest ancestors · 35

Same direction, newest first

  1. 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Management6 Mar
  2. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attacks20 Feb
  3. 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agent18 Feb
  4. 2025Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System17 Feb
  5. 2025AgentGuard: Repurposing Agentic Orchestrator for Safety Evaluation of Tool Orchestration13 Feb
  6. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks12 Feb
  7. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  8. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agents12 Nov
  9. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue6 Nov
  10. 2024Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent2 Nov
  11. 2024Imprompter: Tricking LLM Agents into Improper Tool Use19 Oct
  12. 2024Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents15 Oct
23 more
  1. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents11 Oct
  2. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework11 Oct
  3. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  4. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  5. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul
  6. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent23 Jul
  7. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  8. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities10 Jul
  9. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  10. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
  11. 2024Large Language Model Sentinel: LLM Agent for Adversarial Purification24 May
  12. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  13. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  14. 2024The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative20 Feb
  15. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  16. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  17. 2024LLM Agents can Autonomously Hack Websites6 Feb
  18. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agents2 Feb
  19. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety22 Jan
  20. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  21. 2024Combating Adversarial Attacks with Multi-Agent Debate11 Jan
  22. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
  23. 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents18 Sep

Other references · 49

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  8. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  9. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  10. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  11. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 329
  12. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
37 more
  1. 2023Tool Learning with Foundation Modelscited by 302
  2. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  4. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  5. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  6. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  7. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  8. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  9. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  10. 2025Multi-Agent Risks from Advanced AIcited by 150
  11. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  12. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  13. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  14. 2023Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentencecited by 81
  15. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  16. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  17. 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbationscited by 64
  18. 2024MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debatecited by 54
  19. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  20. 2024Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generationcited by 44
  21. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40
  22. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attackscited by 32
  23. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 31
  24. 2024Agents4PLC: Automating Closed-loop PLC Code Generation and Verification in Industrial Control Systems using LLM-based Agentscited by 25
  25. 2025A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluationscited by 25
  26. 2024Embedding-based classifiers can detect prompt injection attackscited by 24
  27. 2024Prompt Leakage effect and defense strategies for multi-turn LLM interactionscited by 23
  28. 2024Security Attacks on LLM-based Code Completion Toolscited by 15
  29. 2023Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbotscited by 14
  30. 2024On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domainscited by 14
  31. 2024BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguardscited by 13
  32. 2024"Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interactioncited by 9
  33. 2024SLM as Guardian: Pioneering AI Safety with Small Language Modelscited by 7
  34. 2024Good Parenting is all you need -- Multi-agentic LLM Hallucination Mitigationcited by 7
  35. 2025BLAST: A Stealthy Backdoor Leverage Attack against Cooperative Multi-Agent Deep Reinforcement Learning based Systemscited by 7
  36. 2024Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attackscited by 6
  37. 2024Securing Multi-turn Conversational Language Models From Distributed Backdoor Triggerscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 33; 2026: 60.

93 papers.

Sort:

By area:

  1. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 58
  6. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  7. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  8. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  9. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  10. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 23
  11. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 17
  12. 2026AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystemscited by 15
38 more
  1. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 14
  2. 2025A Vision for Access Control in LLM-based Agent Systemscited by 14
  3. 2025Byzantine-Robust Decentralized Coordination of LLM Agentscited by 13
  4. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  5. 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safetycited by 12
  6. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  7. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  8. 2025The Trust Paradox in LLM-Based Multi-Agent Systems: When Collaboration Becomes a Security Vulnerabilitycited by 11
  9. 2026Towards Secure Agent Skills: Architecture, Threat Taxonomy, and Security Analysiscited by 11
  10. 2025Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Servicescited by 10
  11. 2025Code2MCP: Transforming Code Repositories into MCP Servicescited by 10
  12. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 9
  13. 2026OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiencescited by 9
  14. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  15. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 8
  16. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deploymentcited by 8
  17. 2025CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Modelscited by 7
  18. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  19. 2026REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Tracescited by 7
  20. 2025Takedown: How It's Done in Modern Coding Agent Exploitscited by 6
  21. 2026Beyond Similarity: Trustworthy Memory Search for Personal AI Agentscited by 6
  22. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 6
  23. 2025LeechHijack: Covert Computational Resource Exploitation in Intelligent Agent Systemscited by 5
  24. 2026HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generationcited by 5
  25. 2026When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agentscited by 5
  26. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  27. 2025Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustnesscited by 4
  28. 2025Trusted AI Agents in the Cloudcited by 4
  29. 2026Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agentscited by 4
  30. 2026MCPShield: A Security Cognition Layer for Adaptive Trust Calibration in Model Context Protocol Agentscited by 3
  31. 2026You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agentscited by 3
  32. 2026Clawed and Dangerous: Can We Trust Open Agentic Systems?cited by 3
  33. 2026GAMMAF: A Common Framework for Graph-Based Anomaly Monitoring Benchmarking in LLM Multi-Agent Systemscited by 2
  34. 2026Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memorycited by 2
  35. 2026SecureClaw: Clawing Back Control of LLM Agentscited by 2
  36. 2026Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agentscited by 2
  37. 2026Minim: Privacy-Aware Minimal View for Agents via Trusted Local Sanitizationcited by 2
  38. 2026AgentFlow: A Flow-Centric Policy Language and Framework for Securing LLM Agent Systemscited by 2

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025To trust or not to trust: Attention-based Trust Management for LLM Multi-Agent Systems
  2. 2024Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI
  3. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
  4. 2025Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS
  5. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents
  6. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  7. 2026Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
  8. 2026Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted On