# Explainability in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2008.06693 · HTML: https://mlsanity.com/paper/2008.06693/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2020.

## Built on by (most cited first, 30 of 97)

- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions](https://mlsanity.com/paper/2112.11561/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00285/) (2021)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Challenges and Opportunities in Deep Reinforcement Learning with Graph Neural Networks: A Comprehensive review of Algorithms and Applications](https://mlsanity.com/paper/2206.07922/) (2022)
- [Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values](https://mlsanity.com/paper/2110.01307/) (2021)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Formally Explaining Neural Networks within Reactive Systems](https://mlsanity.com/paper/2308.00143/) (2023)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Learning When to Advise Human Decision Makers](https://mlsanity.com/paper/2209.13578/) (2022)
- [GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations](https://mlsanity.com/paper/2302.12689/) (2023)
- [Inroads into Autonomous Network Defence using Explained Reinforcement Learning](https://mlsanity.com/paper/2306.09318/) (2023)
- [State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding](https://mlsanity.com/paper/2309.12482/) (2023)
- [A Survey on AI Assurance](https://mlsanity.com/paper/2111.07505/) (2021)
- [High-Accuracy Model-Based Reinforcement Learning, a Survey](https://mlsanity.com/paper/2107.08241/) (2021)
- [Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2210.11846/) (2022)
- [Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search](https://mlsanity.com/paper/2405.16450/) (2024)
- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [Explainability in reinforcement learning: perspective and position](https://mlsanity.com/paper/2203.11547/) (2022)
- [On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2308.11336/) (2023)
- [Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search](https://mlsanity.com/paper/2212.14849/) (2022)
- [A Closer Look at Reward Decomposition for High-Level Robotic Explanations](https://mlsanity.com/paper/2304.12958/) (2023)
- [Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Driving](https://mlsanity.com/paper/2403.12176/) (2024)
- ["I Don't Think So": Summarizing Policy Disagreements for Agent Comparison](https://mlsanity.com/paper/2102.03064/) (2021)
- [Reinforcement Learning Your Way: Agent Characterization through Policy Regularization](https://mlsanity.com/paper/2201.10003/) (2022)

## Abstract (arXiv)

A large set of the explainable Artificial Intelligence (XAI) literature is emerging on feature relevance techniques to explain a deep neural network (DNN) output or explaining models that ingest image source data. However, assessing how XAI techniques can help understand models beyond classification tasks, e.g. for reinforcement learning (RL), has not been extensively studied. We review recent works in the direction to attain Explainable Reinforcement Learning (XRL), a relatively new subfield of Explainable Artificial Intelligence, intended to be used in general public applications, with diverse audiences, requiring ethical, responsible and trustable algorithms. In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box. We evaluate mainly studies directly linking explainability to RL, and split these into two categories according to the way the explanations are generated: transparent algorithms and post-hoc explainaility. We also review the most prominent XAI works from the lenses of how they could potentially enlighten the further deployment of the latest advances in RL, in the demanding present and future of everyday problems.