# Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applications

arXiv: https://arxiv.org/abs/2207.01911 · HTML: https://mlsanity.com/paper/2207.01911/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 74 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 26 of 26)

- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees](https://mlsanity.com/paper/1807.05887/) (2018)
- [Learning to Perform Physics Experiments via Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01843/) (2016)
- [DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation](https://mlsanity.com/paper/2107.10189/) (2021)
- [Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values](https://mlsanity.com/paper/2110.01307/) (2021)
- [Explaining Reinforcement Learning to Mere Mortals: An Empirical Study](https://mlsanity.com/paper/1903.09708/) (2019)
- [Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario](https://mlsanity.com/paper/2006.13615/) (2020)
- [Machine versus Human Attention in Deep Reinforcement Learning Tasks](https://mlsanity.com/paper/2010.15942/) (2020)
- [Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learning](https://mlsanity.com/paper/2104.02818/) (2021)
- [CDT: Cascading Decision Trees for Explainable Reinforcement Learning](https://mlsanity.com/paper/2011.07553/) (2020)
- [Mo"ET: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learning](https://mlsanity.com/paper/1906.06717/) (2019)
- [DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning](https://mlsanity.com/paper/1909.02982/) (2019)
- [Towards Interpretable-AI Policies Induction using Evolutionary Nonlinear Decision Trees for Discrete Action Systems](https://mlsanity.com/paper/2009.09521/) (2020)
- [Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program](https://mlsanity.com/paper/2106.05506/) (2021)
- [Explaining Deep Reinforcement Learning Agents In The Atari Domain through a Surrogate Model](https://mlsanity.com/paper/2110.03184/) (2021)
- [Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming](https://mlsanity.com/paper/2106.11417/) (2021)
- [Explainable Artificial Intelligence (XAI) for Increasing User Trust in Deep Reinforcement Learning Driven Autonomous Systems](https://mlsanity.com/paper/2106.03775/) (2021)
- [Contrastive Explanations for Comparing Preferences of Reinforcement Learning Agents](https://mlsanity.com/paper/2112.09462/) (2021)
- [Moody Learners -- Explaining Competitive Behaviour of Reinforcement Learning Agents](https://mlsanity.com/paper/2007.16045/) (2020)
- [Robotic Lever Manipulation using Hindsight Experience Replay and Shapley Additive Explanations](https://mlsanity.com/paper/2110.03292/) (2021)
- [Causal versus Marginal Shapley Values for Robotic Lever Manipulation Controlled using Deep Reinforcement Learning](https://mlsanity.com/paper/2111.02936/) (2021)
- [XAI-N: Sensor-based Robot Navigation using Expert Policies and Decision Trees](https://mlsanity.com/paper/2104.10818/) (2021)
- [Explainability of Intelligent Transportation Systems using Knowledge Compilation: a Traffic Light Controller Case](https://mlsanity.com/paper/2007.04916/) (2020)
- [A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals](https://mlsanity.com/paper/2104.06630/) (2021)

## Built on by (most cited first, 6 of 6)

- [A Causal Explainable Guardrails for Large Language Models](https://mlsanity.com/paper/2405.04160/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies](https://mlsanity.com/paper/2508.05433/) (2025)
- [A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making](https://mlsanity.com/paper/2304.10590/) (2023)
- [Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning](https://mlsanity.com/paper/2411.16120/) (2024)
- [Uncovering Latent Phase Structures and Branching Logic in Locomotion Policies: A Case Study on HalfCheetah](https://mlsanity.com/paper/2603.18084/) (2026)

## Abstract (arXiv)

The use of Deep Reinforcement Learning (DRL) schemes has increased dramatically since their first introduction in 2015. Though uses in many different applications are being found, they still have a problem with the lack of interpretability. This has bread a lack of understanding and trust in the use of DRL solutions from researchers and the general public. To solve this problem, the field of Explainable Artificial Intelligence (XAI) has emerged. This entails a variety of different methods that look to open the DRL black boxes, ranging from the use of interpretable symbolic Decision Trees (DT) to numerical methods like Shapley Values. This review looks at which methods are being used and for which applications. This is done to identify which models are the best suited to each application or if a method is being underutilised.