Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
Agentic AI systems built on large language models can plan over multiple steps, use external tools, retain information in memory, and coordinate with other agents. These capabilities make them more useful than static language models, but they also introduce new security and operational risks. Untrusted content from websites, emails, documents, and databases can enter the same context as system instructions; persistent memory can carry compromised information across sessions; and access to external tools can turn an incorrect model response into a consequential real-world action. This article reviews the trustworthiness of agentic AI across five interconnected dimensions: safety and robustness, alignment and human oversight, transparency and auditability, privacy and data governance, and regulatory compliance. It organizes key failure modes, including indirect prompt injection, backdoor triggers, goal misgeneralization, memory contamination, and cross-session data leakage, into a unified taxonomy. It also examines major mitigation approaches, such as instruction hierarchies, context isolation, spotlighting, process-based supervision, constrained tool use, and privacy-preserving memory, while distinguishing techniques supported by empirical evidence from those that remain largely conceptual. Building on this analysis, we introduce the Trustworthy Agent Development Lifecycle (TADL), a six-phase framework covering specification, design, training, evaluation, deployment, and monitoring. For each phase, TADL identifies relevant trust activities, expected evidence, and risk-based decision gates. Although TADL has not yet been empirically validated, it provides a structured foundation for developing and evaluating more secure and accountable agentic systems. The article concludes by identifying gaps in current benchmarks and outlining priorities for future research.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 38 of its 70 references that are in the map. First posted on arXiv 19 Sep 2026.
Builds on
38 of 70 references.
Closest ancestors · 4
Same direction, newest first
- 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems4 Jun
- 2025Unveiling Privacy Risks in LLM Agent Memory17 Feb
- 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents14 Nov
- 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
Other references · 34
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023Let's Verify Step by Stepcited by 3,598
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
22 more
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2018Model Cards for Model Reportingcited by 1,068
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2020Artificial Intelligence, Values and Alignmentcited by 443
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
- 2018Scalable agent alignment via reward modeling: a research directioncited by 438
- 2018AI safety via debatecited by 384
- 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
- 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
- 2021Alignment of Language Agentscited by 167
- 2023Linear Representations of Sentiment in Large Language Modelscited by 157
- 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
- 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
- 2024Safeguarding Large Language Models: A Surveycited by 73
- 2025Overcoming catastrophic forgetting in neural networkscited by 52
- 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
- 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasures
- 2026Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges
- 2026Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- 2026A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems
- 2024Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI
- 2026The Containment Gap: How Deployed Agentic AI Frameworks Fail Public-Facing Safety Requirements
- 2024Whispers in the Machine: Confidentiality in Agentic Systems