# What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes

arXiv: https://arxiv.org/abs/2011.05064 · HTML: https://mlsanity.com/paper/2011.05064/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2020.

## Built on by (most cited first, 20 of 20)

- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Explainable Reinforcement Learning via a Causal World Model](https://mlsanity.com/paper/2305.02749/) (2023)
- [Concept Learning for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.12232/) (2023)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [POETREE: Interpretable Policy Learning with Adaptive Decision Trees](https://mlsanity.com/paper/2203.08057/) (2022)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Explainability Via Causal Self-Talk](https://mlsanity.com/paper/2211.09937/) (2022)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [Contextualized Policy Recovery: Modeling and Interpreting Medical Decisions with Adaptive Imitation Learning](https://mlsanity.com/paper/2310.07918/) (2023)
- [ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policies](https://mlsanity.com/paper/2402.06503/) (2024)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models](https://mlsanity.com/paper/2509.04809/) (2025)
- [Policy Optimization with Sparse Global Contrastive Explanations](https://mlsanity.com/paper/2207.06269/) (2022)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators](https://mlsanity.com/paper/2408.08230/) (2024)
- [RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space](https://mlsanity.com/paper/2410.16517/) (2024)
- [CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systems](https://mlsanity.com/paper/2302.13483/) (2023)
- [Strategically Linked Decisions in Long-Term Planning and Reinforcement Learning](https://mlsanity.com/paper/2505.16833/) (2025)

## Abstract (arXiv)

We present a novel form of explanation for Reinforcement Learning, based around the notion of intended outcome. These explanations describe the outcome an agent is trying to achieve by its actions. We provide a simple proof that general methods for post-hoc explanations of this nature are impossible in traditional reinforcement learning. Rather, the information needed for the explanations must be collected in conjunction with training the agent. We derive approaches designed to extract local explanations based on intention for several variants of Q-function approximation and prove consistency between the explanations and the Q-values learned. We demonstrate our method on multiple reinforcement learning problems, and provide code to help researchers introspecting their RL environments and algorithms.