Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Failure-aware observability diagnoses wasted computation in multi-agent LLM systems before final-answer evaluation can explain what went wrong. We propose a trace-based framework for a three-agent architecture -- orchestrator, search agent, and execution agent -- that converts structured events into online signals for loops, budget pressure, low information gain, and tool instability, then adds offline semantic grounding metrics and selective LLM-as-judge evaluation. On 165 GAIA validation traces under identical caps, 98 runs produce usable final answers and 67 fail or stop without one. Among warned failed runs, 58.1% of tokens are spent after the first warning on average, indicating substantial opportunity for intervention. A 10-task Level-2 pilot uses warnings to diversify search or require evidence, reducing post-warning token fraction from 0.638 in the baseline to 0.304. The results support a layered design: cheap online signals help the orchestrator redirect or halt redundant behavior, while deeper semantic checks identify whether completed answers are grounded enough to trust.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 11 of its 16 references that are in the map. First posted on arXiv 31 May 2026.
Builds on
11 of 16 references.
Closest ancestors · 1
Same direction, newest first
Other references · 10
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 12
- 2026Tiny-Critic RAG: Empowering Agentic Fallback with Parameter-Efficient Small Language Modelscited by 3
- 2026Architecture Matters More Than Scale: A Comparative Study of Retrieval and Memory Augmentation for Financial QA Under SME Compute Constraintscited by 3
- 2021Learning Skill Equivalencies Across Platform Taxonomiescited by 2
- 2026TA-Mem: Tool-Augmented Autonomous Memory Retrieval for LLM in Long-Term Conversational QAcited by 1
- 2026Recursive Multi-Agent Trading System: Iterative Optimized Portfolio Strategy Under Geopolitical Uncertaintycited by 1
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascadecited by 4
- 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluationcited by 1
- 2026Learning Context-Free Grammars for Grammar-Constrained Decoding via Declarative Agentic Programming with Guaranteescited by 0
- 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0
- 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents30 Sep 2026
- 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluation25 Aug 2026
- 2026Learning Context-Free Grammars for Grammar-Constrained Decoding via Declarative Agentic Programming with Guarantees6 Aug 2026
- 2026Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade7 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems
- 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation
- 2026PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors
- 2026When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
- 2026MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems
- 2026Automata from Agent Traces: Failure and Next-Step Prediction
- 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
- 2026ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures