Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training
On the map: LLM agents and tool use → Software engineering coding agents also fits: Agent skill learning and evolution
Abstract (arXiv)
An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 52 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
29 of 52 references.
Closest ancestors · 12
Same direction, newest first
- 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories6 Aug
- 2026AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents21 Jul
- 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?10 Jul
- 2026Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems8 Jul
- 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems26 May
- 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems24 Apr
- 2026AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling22 Mar
- 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectories2 Feb
- 2025The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents5 Nov
- 2025Where LLM Agents Fail and How They can Learn From Failures29 Sep
- 2025TRAIL: Trace Reasoning and Agentic Issue Localization13 May
- 2024Training Software Engineering Agents and Verifiers with SWE-Gym30 Dec
Other references · 17
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025Qwen3 Technical Reportcited by 8,195
- 2017Decoupled Weight Decay Regularizationcited by 1,722
- 2018Datasheets for Datasetscited by 1,371
- 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
- 2023Tool Learning with Foundation Modelscited by 302
- 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
- 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
- 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 61
- 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
- 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
- 2025Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Maskingcited by 16
5 more
- 2025Exploring Expert Failures Improves LLM Agent Tuningcited by 7
- 2026On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignmentcited by 6
- 2026Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectoriescited by 6
- 2026Fantastic Adaptive Taxonomies and How to Use Themcited by 4
- 2026CUADebug: Diagnosing and Repairing Computer-Use Agent Failurescited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems
- 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
- 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systems
- 2026Towards Self-Improving Error Diagnosis in Multi-Agent Systems
- 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems
- 2026Sentry: Learning to Recover from LLM Agent Failures at Test Time
- 2026DeFA: Dependency-Guided Failure Attribution for LLM Agents
- 2026From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis