When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
LLM agent systems increasingly run as long-lived autonomous runtimes: scheduling jobs, calling tools, maintaining memory, and pushing results to humans. We present a longitudinal study of silent failures in one such system: a personal-assistant agent runtime in continuous production since March 2026, with roughly 40 scheduled jobs, 8 LLM providers, a tool-governance proxy, and a knowledge-base memory plane, defended by 4,286 unit tests and 827 governance checks. Over eight weeks we documented 22 incidents with full root-cause postmortems, in which one meta-pattern -- a failure whose error signal never reaches a human in actionable form -- manifested at least 28 times. We derive a five-class, mechanism-oriented taxonomy: (A) environment and platform quirks, (B) design-assumption mismatches, (C) error swallowing and dilution, (D) chained hallucination and fabrication, (E) operational omission and forensic blind spots. Class D is unique to LLM systems and the most dangerous: the system does not merely fail to report an error -- the LLM transforms it into fluent, plausible narrative delivered to the user. We term this fail-plausible: gray failure's differential observability escalated -- the observer is not just blind, it is convincingly lied to by the failure itself. Three findings: about 70% of silent failures were caught by human user-view observation, not tests or audits; a retrospective audit of 15 incidents found 0% ex-ante prevention but 87% regression blocking -- audits are regression engines, not prediction engines; incident latency (13 hours to 60 days) tracks failure mechanism, not code complexity -- the longest-lived failures lived in the seams between components, where no test runs. We describe the resulting defense framework and distill design principles for agent systems whose failures are loud, attributable, and boring. All postmortems and artifacts are public.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 14 references that are in the map. First posted on arXiv 12 Jun 2026.
Builds on
10 of 14 references.
Closest ancestors · 2
Same direction, newest first
- 2025SHIELDA: Structured Handling of Exceptions in LLM-Driven Agentic Workflows11 Aug
- 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
Other references · 8
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2017Chaos Engineeringcited by 19
- 2026AIR: Improving Agent Safety through Incident Responsecited by 8
- 2026Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomycited by 6
- 2025Incident Analysis for AI Agentscited by 5
- 2025Enhancing reliability in AI inference services: An empirical study on real production incidentscited by 5
- 2026A Taxonomy of Runtime Faults in Model Context Protocol Serverscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants
- 2026Silent Failure in LLM Agent Systems: The Entropy Principle and the Inevitable Disorder of Autonomous Agents
- 2026Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
- 2025Where LLM Agents Fail and How They can Learn From Failures
- 2026When Agents Fail to Act: A Diagnostic Framework for Tool Invocation Reliability in Multi-Agent LLM Systems
- 2026Agent Mesh: Reliability Primitives for Non-Idempotent Agent Delegation - Identity Adequacy and Evidence Adequacy
- 2026MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems
- 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents