# AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

arXiv: https://arxiv.org/abs/2602.02475 · HTML: https://mlsanity.com/paper/2602.02475/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 15 of 15)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks](https://mlsanity.com/paper/2411.04468/) (2024)
- [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025)
- [Re-ReST: Reflection-Reinforced Self-Training for Language Agents](https://mlsanity.com/paper/2406.01495/) (2024)
- [AgentGuard: Runtime Verification of AI Agents](https://mlsanity.com/paper/2509.23864/) (2025)
- [Bridging LLM Planning Agents and Formal Methods: A Case Study in Plan Verification](https://mlsanity.com/paper/2510.03469/) (2025)

## Built on by (most cited first, 30 of 60)

- [SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution](https://mlsanity.com/paper/2605.18401/) (2026)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [Wink: Recovering from Misbehaviors in Coding Agents](https://mlsanity.com/paper/2602.17037/) (2026)
- [Auditable Agents](https://mlsanity.com/paper/2604.05485/) (2026)
- [The Art of Building Verifiers for Computer Use Agents](https://mlsanity.com/paper/2604.06240/) (2026)
- [BAGEN: Are LLM Agents Budget-Aware?](https://mlsanity.com/paper/2606.00198/) (2026)
- [AgentAtlas: Beyond Outcome Leaderboards for LLM Agents](https://mlsanity.com/paper/2605.20530/) (2026)
- [DART: Semantic Recoverability for Structured Tool Agents](https://mlsanity.com/paper/2605.23311/) (2026)
- [REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces](https://mlsanity.com/paper/2606.09071/) (2026)
- [AgentRewind: Recoverable Execution for Long-Horizon LLM Agents](https://mlsanity.com/paper/2608.14380/) (2026)
- [ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents](https://mlsanity.com/paper/2609.17523/) (2026)
- [ContextCov: Deriving and Enforcing Executable Constraints from Agent Instruction Files](https://mlsanity.com/paper/2603.00822/) (2026)
- [Willful Disobedience: Automatically Detecting Failures in Agentic Traces](https://mlsanity.com/paper/2603.23806/) (2026)
- [Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.14892/) (2026)
- [Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories](https://mlsanity.com/paper/2606.02060/) (2026)
- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks](https://mlsanity.com/paper/2608.03222/) (2026)
- [Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management](https://mlsanity.com/paper/2605.06472/) (2026)
- [Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents](https://mlsanity.com/paper/2605.08717/) (2026)
- [Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade](https://mlsanity.com/paper/2607.06503/) (2026)
- [Long-Horizon Agent Trajectory Attribution: A Unified Benchmark and Fine-Grained Annotation Framework](https://mlsanity.com/paper/2608.06909/) (2026)
- [Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence](https://mlsanity.com/paper/2608.21156/) (2026)
- [CatchBench: When Can an Agent Failure Be Caught?](https://mlsanity.com/paper/2608.22808/) (2026)
- [interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification](https://mlsanity.com/paper/2602.11202/) (2026)
- [View-oriented Conversation Compiler for Agent Trace Analysis](https://mlsanity.com/paper/2603.29678/) (2026)
- [Cheap Code, Costly Judgment: A Case Study on Governable Agentic Software Engineering](https://mlsanity.com/paper/2607.01087/) (2026)
- [Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming](https://mlsanity.com/paper/2607.11698/) (2026)
- [Online Monitoring and Corrective Steering of Programming Agents](https://mlsanity.com/paper/2608.06701/) (2026)
- [LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectories](https://mlsanity.com/paper/2608.15242/) (2026)
- [Beyond Corrected Memory: Execution Consistency in Multi-Agent Systems](https://mlsanity.com/paper/2610.08101/) (2026)

## Abstract (arXiv)

AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an textitautomated diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.