# Explaining AI Agents Through Execution Traces

arXiv: https://arxiv.org/abs/2609.06063 · HTML: https://mlsanity.com/paper/2609.06063/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 14 of 14)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes](https://mlsanity.com/paper/2603.06847/) (2026)
- [AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems](https://mlsanity.com/paper/2603.29848/) (2026)
- [Willful Disobedience: Automatically Detecting Failures in Agentic Traces](https://mlsanity.com/paper/2603.23806/) (2026)
- [Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement](https://mlsanity.com/paper/2606.27226/) (2026)
- [An Agentic Approach to Generating XAI-Narratives](https://mlsanity.com/paper/2603.20003/) (2026)
- [Responsible Agentic AI Requires Explicit Provenance](https://mlsanity.com/paper/2605.17169/) (2026)
- [From Features to Actions: Explainability in Traditional and Agentic AI Systems](https://mlsanity.com/paper/2602.06841/) (2026)
- [PONTE: Personalized Orchestration for Natural Language Trustworthy Explanations](https://mlsanity.com/paper/2603.06485/) (2026)
- [MATRAG: Multi-Agent Transparent Retrieval-Augmented Generation for Explainable Recommendations](https://mlsanity.com/paper/2604.20848/) (2026)

## Abstract (arXiv)

AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human oversight. This creates a pressing need for reliable and auditable explanations of what an agent did and why. However, traditional Explainable AI (XAI) methods fall short of providing the process-level transparency required for such interactive, multi-step systems, motivating a paradigm shift toward approaches specifically designed for AI Agents. To address this gap, we present a post-hoc XAI framework that transforms a lengthy agent's execution trace into a structured report and a faithful natural-language explanation explicitly grounded in its observable behavior. Because it relies solely on execution traces, the framework applies across different agent architectures, environments, and tasks. Human and automated evaluations across multiple benchmarks and architectures show that our framework produces high-quality, trace-faithful explanations while reliably identifying unsupported claims, unjustified actions, and evidence gaps, outperforming naive LLM-generated explanations.