mlsanityData to 7 Oct 2026

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

arXiv 2605.23989 ↗First posted 17 May 2026cs.AI cs.CL cs.CRcited by 5

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness. This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments: Safety and Robustness, and Privacy and System Security. For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies. Other trustworthiness aspects (value alignment, transparency, fairness, and accountability) are discussed as relevant context rather than parallel chapters. To support consistent comparison and deployment decisions, we consolidate evaluation into a unified metrics-and-benchmarks hub, emphasizing both outcome and process signals (e.g., constraint violations, trace completeness, and adversarial success rates) and offering scenario-to-metric guidance for release gating. We conclude by outlining open challenges such as self-evolving agents, runtime monitoring and verification, privacy-preserving personalization, and the trust-utility trade-off, and present a case study of real-world security failures in open-source agentic systems. Our goal is to serve as a practical reference for researchers and practitioners building trustworthy agentic systems in high-stakes environments.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 110 of its 194 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

110 of 194 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale15 Jan
  2. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safety6 Oct
  3. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents30 Sep
  4. 2025AutoBnB-RAG: Enhancing Multi-Agent Incident Response with Retrieval-Augmented Generation18 Aug
  5. 2025Searching for Privacy Risks in LLM Agents via Simulation14 Aug
  6. 2025WebGuard: Building a Generalizable Guardrail for Web Agents18 Jul
  7. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments2 Jun
  8. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution1 Jun
  9. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasures12 Mar
  10. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents27 Feb
  11. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks12 Feb
  12. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety2 Feb
3 more
  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  2. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agents8 May
  3. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 95

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2018World Modelscited by 1,444
  11. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  13. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  14. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  15. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  16. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  17. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  18. 2017Constrained Policy Optimizationcited by 928
  19. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  20. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  21. 2019Green AIcited by 620
  22. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  23. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  24. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  25. 2017Safe Reinforcement Learning via Shieldingcited by 427
  26. 2023Cognitive Architectures for Language Agentscited by 422
  27. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  28. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  29. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  30. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 253
  31. 2017Inverse Reward Designcited by 246
  32. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  33. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repaircited by 225
  34. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  35. 2020Open Problems in Cooperative AIcited by 167
  36. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  37. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  38. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  2. 2026Memory Has Geometry: Non-Uniform Geometric Memory for Long-Horizon Personalized AIcited by 1
  3. 2026Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agentscited by 0
  4. 2026From AI Technical Debt to Agentic Technical Debt: A Systematic Mapping of Root Causes and Manifestations in Agentic AI Systemscited by 0
  5. 2026RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges
  2. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems
  3. 2026Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
  4. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
  5. 2026Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted On
  6. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
  7. 2026A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems
  8. 2026Securing Agentic AI: From Per-Action Checks to Trajectory Assurance