# Explainable Reinforcement Learning: A Survey

arXiv: https://arxiv.org/abs/2005.06247 · HTML: https://mlsanity.com/paper/2005.06247/

## Key facts

- Cited by 141 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2020.

## Built on by (most cited first, 30 of 141)

- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [What Makes a Good Diffusion Planner for Decision Making?](https://mlsanity.com/paper/2503.00535/) (2025)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values](https://mlsanity.com/paper/2110.01307/) (2021)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Formally Explaining Neural Networks within Reactive Systems](https://mlsanity.com/paper/2308.00143/) (2023)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Explainable Reinforcement Learning via a Causal World Model](https://mlsanity.com/paper/2305.02749/) (2023)
- [Automated Cyber Defence: A Review](https://mlsanity.com/paper/2303.04926/) (2023)
- [Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs](https://mlsanity.com/paper/2301.12950/) (2023)
- [Semantic HELM: A Human-Readable Memory for Reinforcement Learning](https://mlsanity.com/paper/2306.09312/) (2023)
- [Inroads into Autonomous Network Defence using Explained Reinforcement Learning](https://mlsanity.com/paper/2306.09318/) (2023)
- [A Survey on AI Assurance](https://mlsanity.com/paper/2111.07505/) (2021)
- [GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis](https://mlsanity.com/paper/2205.13728/) (2022)
- [Explainable Autonomous Robots: A Survey and Perspective](https://mlsanity.com/paper/2105.02658/) (2021)
- [RLOps: Development Life-cycle of Reinforcement Learning Aided Open RAN](https://mlsanity.com/paper/2111.06978/) (2021)
- [Survey on Fair Reinforcement Learning: Theory and Practice](https://mlsanity.com/paper/2205.10032/) (2022)
- [Interpretable Machine Learning in Physics: A Review](https://mlsanity.com/paper/2503.23616/) (2025)
- [Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2210.11846/) (2022)
- [Improving Human Sequential Decision-Making with Reinforcement Learning](https://mlsanity.com/paper/2108.08454/) (2021)
- [Conformal Prediction Intervals for Markov Decision Process Trajectories](https://mlsanity.com/paper/2206.04860/) (2022)
- [Explainability in reinforcement learning: perspective and position](https://mlsanity.com/paper/2203.11547/) (2022)
- [Toward Policy Explanations for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.12568/) (2022)
- [MIXRTs: Toward Interpretable Multi-Agent Reinforcement Learning via Mixing Recurrent Soft Decision Trees](https://mlsanity.com/paper/2209.07225/) (2022)
- [Explainable Reinforcement Learning via Model Transforms](https://mlsanity.com/paper/2209.12006/) (2022)

## Abstract (arXiv)

Explainable Artificial Intelligence (XAI), i.e., the development of more transparent and interpretable AI models, has gained increased traction over the last few years. This is due to the fact that, in conjunction with their growth into powerful and ubiquitous tools, AI models exhibit one detrimential characteristic: a performance-transparency trade-off. This describes the fact that the more complex a model's inner workings, the less clear it is how its predictions or decisions were achieved. But, especially considering Machine Learning (ML) methods like Reinforcement Learning (RL) where the system learns autonomously, the necessity to understand the underlying reasoning for their decisions becomes apparent. Since, to the best of our knowledge, there exists no single work offering an overview of Explainable Reinforcement Learning (XRL) methods, this survey attempts to address this gap. We give a short summary of the problem, a definition of important terms, and offer a classification and assessment of current XRL methods. We found that a) the majority of XRL methods function by mimicking and simplifying a complex model instead of designing an inherently simple one, and b) XRL (and XAI) methods often neglect to consider the human side of the equation, not taking into account research from related fields like psychology or philosophy. Thus, an interdisciplinary effort is needed to adapt the generated explanations to a (non-expert) human user in order to effectively progress in the field of XRL and XAI in general.