# TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems

arXiv: https://arxiv.org/abs/2605.26563 · HTML: https://mlsanity.com/paper/2605.26563/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 42 of its 68 references that are in the map.
- First posted on arXiv 26 May 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 42)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents](https://mlsanity.com/paper/2505.20411/) (2025)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems](https://mlsanity.com/paper/1709.08071/) (2017)
- [LLM Multi-Agent Systems: Challenges and Open Problems](https://mlsanity.com/paper/2402.03578/) (2024)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?](https://mlsanity.com/paper/2509.03312/) (2025)
- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?](https://mlsanity.com/paper/2511.13646/) (2025)

## Built on by (most cited first, 11 of 11)

- [Failure as a Process: An Anatomy of CLI Coding Agent Trajectories](https://mlsanity.com/paper/2607.09510/) (2026)
- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents](https://mlsanity.com/paper/2608.18398/) (2026)
- [SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents](https://mlsanity.com/paper/2608.05212/) (2026)
- [LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectories](https://mlsanity.com/paper/2608.15242/) (2026)
- [The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks](https://mlsanity.com/paper/2608.16630/) (2026)
- [Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training](https://mlsanity.com/paper/2609.40111/) (2026)
- [Agent System Operations: Categorization, Challenges, and Future Directions](https://mlsanity.com/paper/2606.01581/) (2026)
- [Towards Risk-free AI Agent Deployment](https://mlsanity.com/paper/2608.16411/) (2026)
- [Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase](https://mlsanity.com/paper/2609.29744/) (2026)
- [Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents](https://mlsanity.com/paper/2609.39957/) (2026)

## Abstract (arXiv)

Agentic systems have been widely studied to automate coding tasks such as bug fixing and feature implementation. As these systems increasingly operate on complex codebases, understanding where and why they fail becomes essential for iterative refinement and operational reliability. Existing automated failure diagnosis approaches leverage textittask execution trajectories, yet they struggle with trajectories produced by repository-level coding agents due to two key properties. First, these trajectories are often long, spanning many execution steps, making it difficult for LLMs to track the causal chain of failure over the execution history. Second, these trajectories are laden with noise, containing substantial low-signal observations such as redundant program structures and verbose code context, which can interfere with LLM reasoning. To address these challenges, we propose textitTrajAudit, an automated failure diagnosis framework specifically for trajectories produced by repository-level coding agents. TrajAudit employs an investigator agent supported by two modules: one reduces failure-irrelevant noisy context through semantic saliency folding, and the other derives preliminary diagnostic guidance from test failure reports as prior knowledge to help LLMs focus on likely failure regions. The investigator agent can further invoke tools to inspect folded content on demand, enabling a focused investigation without losing access to the full trajectory context. We also introduce textitRootSE, a benchmark of 102 real-world instances from repository-level coding tasks, each annotated with the earliest decisive error step and a justification. Experiments on RootSE show that TrajAudit outperforms the strongest baselines by 10.8% and 21.6% in exact failure localization accuracy in the with- and without-reference settings, respectively, demonstrating its effectiveness.