mlsanityData to 7 Oct 2026

AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?

arXiv 2509.03312 ↗First posted 3 Sep 2025cs.CL cs.MAcited by 105

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Large Language Model (LLM)-based agentic systems, often comprising multiple models, complex tool invocations, and orchestration protocols, substantially outperform monolithic agents. Yet this very sophistication amplifies their fragility, making them more prone to system failure. Pinpointing the specific agent or step responsible for an error within long execution traces defines the task of agentic system failure attribution. Current state-of-the-art reasoning LLMs, however, remain strikingly inadequate for this challenge, with accuracy generally below 10%. To address this gap, we propose AgenTracer, the first automated framework for annotating failed multi-agent trajectories via counterfactual replay and programmed fault injection, producing the curated dataset TracerTraj. Leveraging this resource, we develop AgenTracer-8B, a lightweight failure tracer trained with multi-granular reinforcement learning, capable of efficiently diagnosing errors in verbose multi-agent interactions. On the Who&When benchmark, AgenTracer-8B outperforms giant proprietary LLMs like Gemini-2.5-Pro and Claude-4-Sonnet by up to 18.18%, setting a new standard in LLM agentic failure attribution. More importantly, AgenTracer-8B delivers actionable feedback to off-the-shelf multi-agent systems like MetaGPT and MaAS with 4.8-14.2% performance gains, empowering self-correcting and self-evolving agentic AI.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 82 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

54 of 82 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems30 Apr
  2. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors7 Apr
  3. 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
  4. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Fly11 Feb
  5. 2024Self-Evolving Multi-Agent Collaboration Networks for Software Development22 Oct
  6. 2024AFlow: Automating Agentic Workflow Generation14 Oct
  7. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul

Other references · 47

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  12. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
35 more
  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  2. 2024Understanding the planning of LLM agents: A surveycited by 414
  3. 2024Automated Design of Agentic Systemscited by 360
  4. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  5. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  6. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 221
  7. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  8. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  9. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  10. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  11. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 164
  12. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  13. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  14. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  15. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  16. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  17. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  18. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 92
  19. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  20. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 80
  21. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  22. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  23. 2025Robin: A multi-agent system for automating scientific discoverycited by 57
  24. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  25. 2025FlowReasoner: Reinforcing Query-Level Meta-Agentscited by 56
  26. 2023Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worldscited by 45
  27. 2024A Scalable Communication Protocol for Networks of Large Language Modelscited by 43
  28. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  29. 2025Optimizing Model Selection for Compound AI Systemscited by 34
  30. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 23
  31. 2024Multi-Agent VQA: Exploring Multi-Agent Foundation Models in Zero-Shot Visual Question Answeringcited by 15
  32. 2024MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verificationcited by 12
  33. 2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11
  34. 2023Can Large Language Models Aid in Annotating Speech Emotional Data? Uncovering New Frontierscited by 10
  35. 2025Improving Model Alignment Through Collective Intelligence of Open-Source LLMScited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 94.

105 papers.

Sort:

By area:

  1. 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 81
  2. 2026SkillClaw: Let Skills Evolve Collectively with Agentic Evolvercited by 76
  3. 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 56
  4. 2026Code as Agent Harnesscited by 40
  5. 2026MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildcited by 35
  6. 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysiscited by 33
  7. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  8. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 23
  9. 2026Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluationcited by 21
  10. 2026From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flawscited by 21
  11. 2026From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systemscited by 17
  12. 2026Agentic Confidence Calibrationcited by 12
38 more
  1. 2026The Last Human-Written Paper: Agent-Native Research Artifactscited by 12
  2. 2025Stop Wasting Your Tokens: Towards Efficient Runtime Multi-Agent Systemscited by 11
  3. 2025How Far Are We from Genuinely Useful Deep Research Agents?cited by 11
  4. 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systemscited by 11
  5. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 11
  6. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  7. 2025Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflowscited by 10
  8. 2026From Confident Closing to Silent Failure: Characterizing False Success in LLM Agentscited by 10
  9. 2026FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Searchcited by 9
  10. 2026AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Tracescited by 9
  11. 2025Build Your Personalized Research Group: A Multiagent Framework for Continual and Interactive Science Automationcited by 8
  12. 2026The Why Behind the Action: Unveiling Internal Drivers via Agentic Attributioncited by 8
  13. 2026AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systemscited by 8
  14. 2026Shepherd: Enabling Programmable Meta-Agents via Reversible Agentic Execution Tracescited by 8
  15. 2025Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphscited by 7
  16. 2026REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Tracescited by 7
  17. 2026AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agentscited by 7
  18. 2025Agentic Services Computingcited by 6
  19. 2026Efficient Failure Management for Multi-Agent Systems with Reasoning Trace Representationcited by 5
  20. 2026MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signalscited by 5
  21. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systemscited by 5
  22. 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systemscited by 5
  23. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 5
  24. 2026What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agentscited by 5
  25. 2026Tracing Agentic Failure from the Flow of Successcited by 5
  26. 2026AI Organizations are More Effective but Less Aligned than Individual Agentscited by 4
  27. 2026PIVOT: Bridging Planning and Execution in LLM Agents via Trajectory Refinementcited by 4
  28. 2026Do Coding Agents Understand Least-Privilege Authorization?cited by 4
  29. 2025FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipelinecited by 3
  30. 2026Responsible Agentic AI Requires Explicit Provenancecited by 3
  31. 2026CASPIAN: Online Detection and Attribution of Cascade Attacks in LLM Multi-Agent Systems via Cross-Channel Causal Monitoringcited by 3
  32. 2026When Planning Fails Despite Correct Execution: On Epistemic Calibration for LLM-Based Multi-Agent Systemscited by 3
  33. 2026MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubricscited by 2
  34. 2026ProMAS: Proactive Error Forecasting for Multi-Agent Systems Using Markov Transition Dynamicscited by 2
  35. 2026Towards Self-Improving Error Diagnosis in Multi-Agent Systemscited by 2
  36. 2026What Happens Inside Agent Memory? Circuit Analysis from Emergence to Diagnosiscited by 2
  37. 2026DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agentscited by 2
  38. 2026QUIVER: A Formal Framework for Quantifying Perturbation Propagation and Bifurcation in Compound AI Systemscited by 2

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Where LLM Agents Fail and How They can Learn From Failures
  2. 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
  3. 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems
  4. 2025How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations
  5. 2026Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems
  6. 2026Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training
  7. 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
  8. 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break