Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves Opus-4.8's F1 score by 29%, from 0.471 to 0.608. More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 41 references that are in the map. First posted on arXiv 11 Sep 2026.
Builds on
31 of 41 references.
Closest ancestors · 5
Same direction, newest first
- 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories6 Aug
- 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?10 Jul
- 2026DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks8 Jul
- 2026From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization8 Jul
- 2025TRAIL: Trace Reasoning and Agentic Issue Localization13 May
Other references · 26
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
- 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
14 more
- 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
- 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
- 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
- 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
- 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
- 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 28
- 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 21
- 2025Challenging the Evaluator: LLM Sycophancy Under User Rebuttalcited by 18
- 2026Remember When It Matters: Proactive Memory Agent for Long-Horizon Agentscited by 10
- 2026When Agents Commit Too Soon: Diagnosing Premature Commitment in LLM Agentscited by 9
- 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Taskscited by 9
- 2026Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failurescited by 6
- 2026Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agentscited by 5
- 2026Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judgescited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation
- 2026MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signals
- 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
- 2026From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis
- 2026DeFA: Dependency-Guided Failure Attribution for LLM Agents
- 2026FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search
- 2026OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
- 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems