mlsanityData to 14 Oct 2026

REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces

arXiv 2606.09071 ↗First posted 8 Jun 2026cs.AIcited by 7

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Large language model (LLM) agents now solve complex tasks through long plan-and-execution traces, yet the ability to locate errors in a completed traces still lags far behind, especially in the silent failure regime. Existing approaches predict suspect steps via classifiers or LLM judges, or recover correct answers via retry, but none feed the intervention outcome back to refine the attribution itself. We propose methodname, a method that closes this gap by diagnosing a candidate error step, testing it through controlled replay with a diagnosis-specific patch, and using the verified outcome flip as contrastive evidence to refine the final attribution. Across four localization benchmarks spanning multi-hop reasoning across domains, methodname achieves the highest localization accuracy among same-auditor methods across all four benchmarks, with the largest gains on structured tool-use traces, while providing actionable localization even when ground-truth answers are unavailable.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 45 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

34 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectories2 Feb
  2. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?3 Sep
  3. 2025TRAIL: Trace Reasoning and Agentic Issue Localization13 May
  4. 2024AgentOps: Enabling Observability of LLM Agents8 Nov

Other references · 30

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  10. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  12. 2023Teaching Large Language Models to Self-Debugcited by 988
18 more
  1. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  2. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  3. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  4. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  5. 2025Survey on Evaluation of LLM-based Agentscited by 204
  6. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  7. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 127
  8. 2025Process Reward Models That Thinkcited by 118
  9. 2025Agentic Large Language Models, a surveycited by 108
  10. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  11. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 85
  12. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 77
  13. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  14. 2024General Purpose Verification for Chain of Thought Promptingcited by 17
  15. 2026TrajAD: Trajectory Anomaly Detection for Trustworthy LLM Agentscited by 15
  16. 2024Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agentscited by 8
  17. 2025LLMs cannot spot math errors, even when allowed to peek into the solutioncited by 7
  18. 2026Structure Enables Effective Self-Localization of Errors in LLMscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?cited by 1
  2. 2026DeFA: Dependency-Guided Failure Attribution for LLM Agentscited by 1
  3. 2026Chronicle: Cut-Point Replay for Regression Testing of LLM Agentscited by 0
  4. 2026CausalLoss-Fin: Attributing Financial-Agent Loss to Decisions and Infrastructure Faultscited by 0
  5. 2026ReplayLens: Auditing Agents' Use of Outcomescited by 0
  6. 2026Towards Reliable AI Data Scientists: Data Agents with Workflow Harnessescited by 0
  7. 2026Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search
  2. 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems
  3. 2026Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems
  4. 2026PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors
  5. 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents
  6. 2026DAAF: From Failure Localization to Editable System Assets in LLM Agents
  7. 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
  8. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems