mlsanityData to 7 Oct 2026

A Survey of Explainable Reinforcement Learning

arXiv 2202.08434 ↗First posted 17 Feb 2022cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Explainable reinforcement learning (XRL) is an emerging subfield of explainable machine learning that has attracted considerable attention in recent years. The goal of XRL is to elucidate the decision-making process of learning agents in sequential decision-making settings. In this survey, we propose a novel taxonomy for organizing the XRL literature that prioritizes the RL setting. We overview techniques according to this taxonomy. We point out gaps in the literature, which we use to motivate and outline a roadmap for future work.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 42 of its 74 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

42 of 74 references.

References · 42

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  3. 2016The Mythos of Model Interpretabilitycited by 716
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  5. 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
  6. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  7. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  8. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  9. 2017Visualizing and Understanding Atari Agentscited by 141
  10. 2020Explainable Reinforcement Learning: A Surveycited by 139
  11. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
  12. 2020Explainability in Deep Reinforcement Learningcited by 96
30 more
  1. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  2. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  3. 2016Graying the black box: Understanding DQNscited by 68
  4. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  5. 2020Neuroevolution of Self-Interpretable Agentscited by 48
  6. 2017Interpretable Policies for Reinforcement Learning by Genetic Programmingcited by 47
  7. 2018Establishing Appropriate Trust via Critical Statescited by 47
  8. 2020Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitionscited by 41
  9. 2018Transparency and Explanation in Deep Reinforcement Learning Neural Networkscited by 40
  10. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 38
  11. 2017Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanationscited by 37
  12. 2018Learning Finite State Representations of Recurrent Policy Networkscited by 37
  13. 2019Generation of Policy-Level Explanations for Reinforcement Learningcited by 37
  14. 2019Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulationcited by 33
  15. 2021Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methodscited by 31
  16. 2019Exploring Computational User Models for Agent Policy Summarizationcited by 29
  17. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  18. 2018Towards Better Interpretability in Deep Q-Networkscited by 22
  19. 2019Explaining Reinforcement Learning to Mere Mortals: An Empirical Studycited by 21
  20. 2020What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomescited by 20
  21. 2016Particle Swarm Optimization for Generating Interpretable Fuzzy Reinforcement Learning Policiescited by 19
  22. 2020Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenariocited by 19
  23. 2020TripleTree: A Versatile Interpretable Representation of Black Box Agents and their Environmentscited by 18
  24. 2019Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agentscited by 15
  25. 2020Re-understanding Finite-State Representations of Recurrent Policy Networkscited by 15
  26. 2020Contrastive Explanations for Reinforcement Learning via Embedded Self Predictionscited by 13
  27. 2018Unsupervised Video Object Segmentation for Deep Reinforcement Learningcited by 11
  28. 2021Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learningcited by 9
  29. 2019Visual Rationalizations in Deep Reinforcement Learning for Atari Gamescited by 8
  30. 2020Self-Supervised Discovering of Interpretable Features for Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 9; 2025: 6; 2026: 2.

31 papers.

Sort:

By area:

  1. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  2. 2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 41
  3. 2023Neurosymbolic Reinforcement Learning and Planning: A Surveycited by 24
  4. 2023Automated Cyber Defence: A Reviewcited by 19
  5. 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learningcited by 18
  6. 2024End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanationscited by 17
  7. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
  8. 2022MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learningcited by 12
  9. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  10. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
  11. 2024Bootstrapping Cognitive Agents with a Large Language Modelcited by 9
  12. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
19 more
  1. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  2. 2022Experiential Explanations for Reinforcement Learningcited by 5
  3. 2024Optimizing Interpretable Decision Tree Policies for Reinforcement Learningcited by 4
  4. 2023Causal State Distillation for Explainable Reinforcement Learningcited by 2
  5. 2024Scalable Interactive Machine Learning for Future Command and Controlcited by 2
  6. 2024XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniquescited by 2
  7. 2022Multi-Resolution Online Deterministic Annealing: A Hierarchical and Progressive Learning Architecturecited by 1
  8. 2023Limits of Actor-Critic Algorithms for Decision Tree Policies Learning in IBMDPscited by 1
  9. 2023Utilizing Explainability Techniques for Reinforcement Learning Model Assurancecited by 1
  10. 2022Explaining Agent's Decision-making in a Hierarchical Reinforcement Learning Scenariocited by 0
  11. 2023Personalized Decision Supports based on Theory of Mind Modeling and Explainable Reinforcement Learningcited by 0
  12. 2024Backward explanations via redefinition of predicatescited by 0
  13. 2024BaTCAVe: Trustworthy Explanations for Robot Behaviorscited by 0
  14. 2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0
  15. 2025Exploring Explainable Multi-agent MCTS-minimax Hybrids in Board Game Using Process Miningcited by 0
  16. 2025"What are my options?": Explaining RL Agents with Diverse Near-Optimal Alternatives (Extended)cited by 0
  17. 2025STACHE: Local Black-Box Explanations for Reinforcement Learning Policiescited by 0
  18. 2026Evidence of an Emergent "Self" in Continual Robot Learningcited by 0
  19. 2026Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learningcited by 0