mlsanityData to 7 Oct 2026

Local Explanations for Reinforcement Learning

arXiv 2202.03597 ↗First posted 8 Feb 2022cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Many works in explainable AI have focused on explaining black-box classification models. Explaining deep reinforcement learning (RL) policies in a manner that could be understood by domain users has received much less attention. In this paper, we propose a novel perspective to understanding RL policies based on identifying important states from automatically learned meta-states. The key conceptual difference between our approach and many previous ones is that we form meta-states based on locality governed by the expert policy dynamics rather than based on similarity of actions, and that we do not assume any particular knowledge of the underlying topology of the state space. Theoretically, we show that our algorithm to find meta-states converges and the objective that selects important states from each meta-state is submodular leading to efficient high quality greedy selection. Experiments on four domains (four rooms, door-key, minipacman, and pong) and a carefully conducted user study illustrate that our perspective leads to better understanding of the policy. We conjecture that this is a result of our meta-states being more intuitive in that the corresponding important states are strong indicators of tractable intermediate goals that are easier for humans to interpret and follow.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 28 of its 55 references that are in the map. First posted on arXiv 8 Feb 2022.

Builds on

28 of 55 references.

References · 28

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2016OpenAI Gymcited by 1,684
  4. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  5. 2019Attention is not Explanationcited by 916
  6. 2016The Mythos of Model Interpretabilitycited by 716
  7. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  8. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  9. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  10. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  11. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  12. 2017Visualizing and Understanding Atari Agentscited by 141
16 more
  1. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  2. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  3. 2018Hierarchical interpretations for neural network predictionscited by 66
  4. 2018Establishing Appropriate Trust via Critical Statescited by 47
  5. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 38
  6. 2018Learning Finite State Representations of Recurrent Policy Networkscited by 37
  7. 2019Generation of Policy-Level Explanations for Reinforcement Learningcited by 37
  8. 2020Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Mapscited by 30
  9. 2019Exploring Computational User Models for Agent Policy Summarizationcited by 29
  10. 2019Learning from Trajectories via Subgoal Discoverycited by 27
  11. 2015State of the Art Control of Atari Games Using Shallow Reinforcement Learningcited by 23
  12. 2019Successor Options: An Option Discovery Framework for Reinforcement Learningcited by 23
  13. 2020What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomescited by 20
  14. 2019Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agentscited by 15
  15. 2020Re-understanding Finite-State Representations of Recurrent Policy Networkscited by 15
  16. 2017TIP: Typifying the Interpretability of Procedurescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 1; 2025: 0; 2026: 2.

5 papers.

  1. 2022MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learningcited by 12
  2. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  3. 2024Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learningcited by 2
  4. 2026Task-Induced Representational Invariances Depend on Learning Objective in Deep RLcited by 1
  5. 2026Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entrycited by 0