mlsanityData to 7 Oct 2026

What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes

arXiv 2011.05064 ↗First posted 10 Nov 2020cs.AI cs.CY cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

We present a novel form of explanation for Reinforcement Learning, based around the notion of intended outcome. These explanations describe the outcome an agent is trying to achieve by its actions. We provide a simple proof that general methods for post-hoc explanations of this nature are impossible in traditional reinforcement learning. Rather, the information needed for the explanations must be collected in conjunction with training the agent. We derive approaches designed to extract local explanations based on intention for several variants of Q-function approximation and prove consistency between the explanations and the Q-values learned. We demonstrate our method on multiple reinforcement learning problems, and provide code to help researchers introspecting their RL environments and algorithms.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 6; 2023: 6; 2024: 4; 2025: 3.

20 papers.

Sort:

By area:

  1. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  2. 2022A Survey of Explainable Reinforcement Learningcited by 31
  3. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  4. 2023Concept Learning for Interpretable Multi-Agent Reinforcement Learningcited by 18
  5. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  6. 2022POETREE: Interpretable Policy Learning with Adaptive Decision Treescited by 14
  7. 2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
  8. 2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
  9. 2022Explainability Via Causal Self-Talkcited by 6
  10. 2022Local Explanations for Reinforcement Learningcited by 5
  11. 2023Contextualized Policy Recovery: Modeling and Interpreting Medical Decisions with Adaptive Imitation Learningcited by 4
  12. 2024ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policiescited by 4
8 more
  1. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  2. 2022Policy Optimization with Sparse Global Contrastive Explanationscited by 2
  3. 2024Explainable Interface for Human-Autonomy Teaming: A Surveycited by 2
  4. 2024Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimatorscited by 2
  5. 2024RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Spacecited by 2
  6. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Modelscited by 2
  7. 2023CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systemscited by 0
  8. 2025Strategically Linked Decisions in Long-Term Planning and Reinforcement Learningcited by 0