mlsanityData to 7 Oct 2026

Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attribution

arXiv 2510.04886 ↗First posted 6 Oct 2025cs.AI cs.MAcited by 25

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Error attribution in Large Language Model (LLM) multi-agent systems presents a significant challenge in debugging and improving collaborative AI systems. Current approaches to pinpointing agent and step level failures in interaction traces - whether using all-at-once evaluation, step-by-step analysis, or binary search - fall short when analyzing complex patterns, struggling with both accuracy and consistency. We present ECHO (Error attribution through Contextual Hierarchy and Objective consensus analysis), a novel algorithm that combines hierarchical context representation, objective analysis-based evaluation, and consensus voting to improve error attribution accuracy. Our approach leverages a positional-based leveling of contextual understanding while maintaining objective evaluation criteria, ultimately reaching conclusions through a consensus mechanism. Experimental results demonstrate that ECHO outperforms existing methods across various multi-agent interaction scenarios, showing particular strength in cases involving subtle reasoning errors and complex interdependencies. Our findings suggest that leveraging these concepts of structured, hierarchical context representation combined with consensus-based objective decision-making, provides a more robust framework for error attribution in multi-agent systems.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 48 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

31 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems30 Apr

Other references · 30

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  10. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  11. 2023GPTScore: Evaluate as You Desirecited by 357
  12. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
18 more
  1. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  2. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  3. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 169
  4. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  5. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  6. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
  7. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  8. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  9. 2024Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligencecited by 80
  10. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learningcited by 74
  11. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  12. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  13. 2025Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Modelscited by 27
  14. 2022Contrastive Error Attribution for Finetuned Language Modelscited by 12
  15. 2025Benchmarking LLMs' Swarm intelligencecited by 8
  16. 2024Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generationcited by 6
  17. 2025Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratingscited by 5
  18. 2025Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Frameworkcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 24.

25 papers.

Sort:

By area:

  1. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 23
  2. 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systemscited by 22
  3. 2026From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systemscited by 17
  4. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  5. 2026FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Searchcited by 9
  6. 2026MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signalscited by 5
  7. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systemscited by 5
  8. 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systemscited by 5
  9. 2026Tracing Agentic Failure from the Flow of Successcited by 5
  10. 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectoriescited by 5
  11. 2026FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environmentscited by 4
  12. 2026Conformal Agent Error Attributioncited by 3
13 more
  1. 2026EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systemscited by 3
  2. 2026Towards Self-Improving Error Diagnosis in Multi-Agent Systemscited by 2
  3. 2026Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systemscited by 2
  4. 2026StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systemscited by 2
  5. 2026Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systemscited by 1
  6. 2026CockpitHAT: Dependency-Graph-Driven Hierarchical Attribution for Embodied Multi-Agent Cockpitscited by 1
  7. 2026LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectoriescited by 1
  8. 2026Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inferencecited by 0
  9. 2026Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attributioncited by 0
  10. 2026Adaptive Influence Graphs for Failure Attribution in Multi-Agent Systemscited by 0
  11. 2026Detect Before You Attribute: Cascade Failure Attribution for Multi-Agent Systemscited by 0
  12. 2026Know the Shape, Find the Fault: Topology-Conditioned Diagnosis of Multi-Agent LLM Failurescited by 0
  13. 2026Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems
  2. 2026Finding Where the Buck Stops: An Automated Failure Attribution-Based Reflection Framework for Multi-Agent Collaboration
  3. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systems
  4. 2026Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation
  5. 2025CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems
  6. 2026ASCon: A Direction-Aware Reciprocal Agent--Step Contextualization Model for Failure Attribution in Multi-Agent Systems
  7. 2026Towards a Science of Collective AI: LLM-based Multi-Agent Systems Need a Transition from Blind Trial-and-Error to Rigorous Science
  8. 2025Abduct, Act, Predict: Scaffolding Causal Inference for Automated Failure Attribution in Multi-Agent Systems