mlsanityData to 7 Oct 2026

TRAIL: Trace Reasoning and Agentic Issue Localization

arXiv 2505.08638 ↗First posted 13 May 2025cs.AI cs.CLcited by 98

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Current evaluation methods depend on manual, domain-specific human analysis of lengthy workflow traces - an approach that does not scale with the growing complexity and volume of agentic outputs. Error analysis in these settings is further complicated by the interplay of external tool outputs and language model reasoning, making it more challenging than traditional software debugging. In this work, we (1) articulate the need for robust and dynamic evaluation methods for agentic workflow traces, (2) introduce a formal taxonomy of error types encountered in agentic systems, and (3) present a set of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and grounded in established agentic benchmarks. To ensure ecological validity, we curate traces from both single and multi-agent systems, focusing on real-world applications such as software engineering and open-world information retrieval. Our evaluations reveal that modern long context LLMs perform poorly at trace debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL. Our dataset and code are made publicly available to support and accelerate future research in scalable evaluation for agentic workflows.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 73 of its 212 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

73 of 212 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems12 Apr
  2. 2025ToolFuzz -- Automated Agent Tool Testing6 Mar
  3. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation18 Feb
  4. 2024ACPBench: Reasoning about Action, Change, and Planning8 Oct
  5. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation1 Aug
  6. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents7 Jun

Other references · 67

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  10. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  11. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  12. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
38 more
  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  5. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  6. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  7. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  9. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  10. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  11. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  12. 2023GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarkingcited by 209
  13. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  14. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  15. 2025Survey on Evaluation of LLM-based Agentscited by 190
  16. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  17. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  18. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  19. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  20. 2020BLEU might be Guilty but References are not Innocentcited by 109
  21. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 100
  22. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 95
  23. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  24. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  25. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
  26. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 83
  27. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  28. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  29. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  30. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  31. 2025Interactive Debugging and Steering of Multi-Agent AI Systemscited by 68
  32. 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
  33. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 66
  34. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  35. 2024STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Basescited by 53
  36. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  37. 2023LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystemcited by 35
  38. 2024LLMParser: An Exploratory Study on Using Large Language Models for Log Parsingcited by 32

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 81.

98 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  3. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  5. 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysiscited by 33
  6. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  7. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  8. 2025When Agents go Astray: Course-Correcting SWE Agents with PRMscited by 25
  9. 2025Process-Centric Analysis of Agentic Software Systemscited by 24
  10. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 23
  11. 2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systemscited by 19
  12. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systemscited by 19
38 more
  1. 2025Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inferencecited by 18
  2. 2026AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluationcited by 13
  3. 2025Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systemscited by 12
  4. 2026Wink: Recovering from Misbehaviors in Coding Agentscited by 11
  5. 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systemscited by 11
  6. 2026AgentStepper: Interactive Debugging of Software Development Agentscited by 8
  7. 2026How to Interpret Agent Behaviorcited by 8
  8. 2025What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignmentcited by 7
  9. 2026REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Tracescited by 7
  10. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
  11. 2026MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observabilitycited by 6
  12. 2026When Agents Fail: A Comprehensive Study of Bugs in LLM Agents with Automated Labelingcited by 6
  13. 2026ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Modelscited by 6
  14. 2026Willful Disobedience: Automatically Detecting Failures in Agentic Tracescited by 6
  15. 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systemscited by 6
  16. 2026Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectoriescited by 6
  17. 2026Efficient Failure Management for Multi-Agent Systems with Reasoning Trace Representationcited by 5
  18. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systemscited by 5
  19. 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generationcited by 5
  20. 2026What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agentscited by 5
  21. 2026Tracing Agentic Failure from the Flow of Successcited by 5
  22. 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectoriescited by 5
  23. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  24. 2026PIVOT: Bridging Planning and Execution in LLM Agents via Trajectory Refinementcited by 4
  25. 2026Delayed Verification Destabilizes Multi-Agent LLM Belief: Instability Thresholds and Optimal Corrector Placementcited by 4
  26. 2026Fantastic Adaptive Taxonomies and How to Use Themcited by 4
  27. 2026How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Taskscited by 4
  28. 2026Think Locally, Explain Globally: Graph-Guided LLM Investigations via Local Reasoning and Belief Propagationcited by 3
  29. 2026Sketch2Simulation: Automating Flowsheet Generation via Multi Agent Large Language Modelscited by 3
  30. 2026Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systemscited by 3
  31. 2026EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systemscited by 3
  32. 2025From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineeringcited by 2
  33. 2026Towards Structured, State-Aware, and Execution-Grounded Reasoning for Software Engineering Agentscited by 2
  34. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agentscited by 2
  35. 2026StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systemscited by 2
  36. 2026Knowledge-Based Zero-Replay Debugging of Multi-Agent LLM Tracescited by 2
  37. 2026CatchBench: When Can an Agent Failure Be Caught?cited by 2
  38. 2026Resolving Java Code Repository Issues with iSWE Agentcited by 1

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production
  2. 2026TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces
  3. 2026AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
  4. 2026AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
  5. 2026Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents
  6. 2026TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces
  7. 2025XAgen: An Explainability Tool for Identifying and Correcting Failures in Multi-Agent Workflows
  8. 2026TraceCompiler: Skill-Guided Mining and Compilation of LLM Agent Traces into Mostly Deterministic Workflows