FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search
On the map: LLM reasoning → Reasoning for agents, planning and code
also in 1 more
Abstract (arXiv)
LLM-based agents increasingly solve complex tasks through long trajectories involving reasoning steps, tool calls, and inter-agent communication. However, when these agents fail, it is often unclear which agent caused the failure and which step introduced the decisive error. This attribution problem is challenging because mistakes can propagate across the trajectory: later actions may appear incorrect, but only because they depend on an earlier corrupted state. Therefore, failure attribution cannot be treated as independent step-level classification. We propose FALAT, a diagnostic framework for failure attribution in LLM agent trajectories. FALAT frames attribution as a dependency-guided search problem. It first constructs an expectation of how the task should be solved and uses this expectation to identify suspicious regions in the trajectory. It then traces dependencies among decisions, tool outputs, and agent messages to distinguish error-introducing steps from steps that merely inherit or propagate prior mistakes. Finally, FALAT evaluates whether correcting a candidate step would be sufficient to recover the expected outcome, allowing it to identify both the responsible agent and the decisive failure step. We evaluate FALAT on the Who&When benchmark, which includes both algorithm-generated and hand-crafted multi-agent failure trajectories. The results show that FALAT consistently improves responsible-agent and decisive-step attribution. Its best configurations achieve 46.0% step-level accuracy on algorithm-generated trajectories and 29.1% on the more challenging hand-crafted trajectories, outperforming specialized attribution baselines and direct prompting with standalone LLMs. These findings suggest that dependency-aware reasoning is essential for reliable failure diagnosis in LLM agent systems.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 33 of its 39 references that are in the map. First posted on arXiv 30 May 2026.
Builds on
33 of 39 references.
Closest ancestors · 3
Same direction, newest first
- 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories23 Jun
- 2025Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents19 May
- 2024AFlow: Automating Agentic Workflow Generation14 Oct
Other references · 30
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2023ChatDev: Communicative Agents for Software Developmentcited by 893
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
- 2024Automated Design of Agentic Systemscited by 360
- 2024TextGrad: Automatic "Differentiation" via Textcited by 250
- 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 221
18 more
- 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
- 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
- 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
- 2025Where LLM Agents Fail and How They can Learn From Failurescited by 118
- 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
- 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 80
- 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
- 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 46
- 2025UTBoost: Rigorous Evaluation of Coding Agents on SWE-Benchcited by 44
- 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
- 2025Optimizing Model Selection for Compound AI Systemscited by 34
- 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysiscited by 33
- 2025Adaptive Graph Pruning for Multi-Agent Communicationcited by 28
- 2025Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attributioncited by 25
- 2025GraphTracer: Graph-Guided Failure Tracing in LLM Agents for Robust Multi-Turn Deep Searchcited by 22
- 2026From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systemscited by 17
- 2025Agent-SAMA: State-Aware Mobile Assistantcited by 12
- 2024Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localizationcited by 3
Built on by
Citing papers per year: 2026: 9.
9 papers.
- 2026GRADE: Graph Representation of LLM Agent Dependency and Executioncited by 4
- 2026How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Taskscited by 4
- 2026SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agentscited by 1
- 2026Online Monitoring and Corrective Steering of Programming Agentscited by 1
- 2026Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independencecited by 1
- 2026LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectoriescited by 1
- 2026Adaptive Influence Graphs for Failure Attribution in Multi-Agent Systemscited by 0
- 2026Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agentscited by 0
- 2026ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systemscited by 0
- 2026ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems8 Oct 2026
- 2026Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents30 Sep 2026
- 2026Adaptive Influence Graphs for Failure Attribution in Multi-Agent Systems25 Aug 2026
- 2026LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectories15 Aug 2026
- 2026How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks14 Aug 2026
- 2026Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence13 Aug 2026
- 2026Online Monitoring and Corrective Steering of Programming Agents7 Aug 2026
- 2026SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents5 Aug 2026
- 2026GRADE: Graph Representation of LLM Agent Dependency and Execution22 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026DeFA: Dependency-Guided Failure Attribution for LLM Agents
- 2026StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems
- 2026Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent Trajectories
- 2026Tracing Agentic Failure from the Flow of Success
- 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
- 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems
- 2026Detect Before You Attribute: Cascade Failure Attribution for Multi-Agent Systems
- 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories