mlsanityData to 7 Oct 2026

XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights

arXiv 2603.05941 ↗First posted 6 Mar 2026cs.SE cs.AIcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Large Language Model (LLM)-based coding agents show promise in automating software development tasks, yet they frequently fail in ways that are difficult for developers to understand and debug. While general-purpose LLMs like GPT can provide ad-hoc explanations of failures, raw execution traces remain challenging to interpret even for experienced developers. We present a systematic explainable AI (XAI) approach that transforms raw agent execution traces into structured, human-interpretable explanations. Our method consists of three key components: (1) a domain-specific failure taxonomy derived from analyzing real agent failures, (2) an automatic annotation system that classifies failures using defined annotation schema, (3) a hybrid explanation generator that produces visual execution flows, natural language explanations, and actionable recommendations. Through a user study with 20 participants (10 technical, 10 non-technical), we demonstrate that our approach enables users to identify failure root causes 2.8 times faster and propose correct fixes with 73% higher accuracy compared to raw execution traces. Importantly, our structured approach outperforms ad-hoc state of the art models explanations by providing consistent, domain-specific insights with integrated visualizations. Our work establishes a framework for systematic agent failure analysis, addressing the critical need for interpretable AI systems in software development workflows

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 7 of its 14 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

7 of 14 references.

References · 7

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2021Perfection Not Required? Human-AI Partnerships in Code Translationcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  2. 2026ParallelPilot: Supporting Coordination and Monitoring in Parallel AI Codingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
  2. 2025XAgen: An Explainability Tool for Identifying and Correcting Failures in Multi-Agent Workflows
  3. 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems
  4. 2026TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code
  5. 2025Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories
  6. 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
  7. 2025Where LLM Agents Fail and How They can Learn From Failures
  8. 2026Explaining AI Agents Through Execution Traces