mlsanityData to 14 Oct 2026

Explainable Deep Reinforcement Learning: State of the Art and Challenges

arXiv 2301.09937 ↗First posted 24 Jan 2023cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Interpretability, explainability and transparency are key issues to introducing Artificial Intelligence methods in many critical domains: This is important due to ethical concerns and trust issues strongly connected to reliability, robustness, auditability and fairness, and has important consequences towards keeping the human in the loop in high levels of automation, especially in critical cases for decision making, where both (human and the machine) play important roles. While the research community has given much attention to explainability of closed (or black) prediction boxes, there are tremendous needs for explainability of closed-box methods that support agents to act autonomously in the real world. Reinforcement learning methods, and especially their deep versions, are such closed-box methods. In this article we aim to provide a review of state of the art methods for explainable deep reinforcement learning methods, taking also into account the needs of human operators - i.e., of those that take the actual and critical decisions in solving real-world problems. We provide a formal specification of the deep reinforcement learning explainability problems, and we identify the necessary components of a general explainable reinforcement learning framework. Based on these, we provide a comprehensive review of state of the art methods, categorizing them in classes according to the paradigm they follow, the interpretable models they use, and the surface representation of explanations provided. The article concludes identifying open questions and important challenges.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 41 of its 74 references that are in the map. First posted on arXiv 24 Jan 2023.

Builds on

41 of 74 references.

References · 41

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  5. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  6. 2016The Mythos of Model Interpretabilitycited by 721
  7. 2015Policy Distillationcited by 440
  8. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
  9. 2019Interpretable machine learning: definitions, methods, and applicationscited by 305
  10. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
  11. 2017Distilling a Neural Network Into a Soft Decision Treecited by 236
  12. 2018Programmatically Interpretable Reinforcement Learningcited by 193
29 more
  1. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  2. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  3. 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 166
  4. 2017Visualizing and Understanding Atari Agentscited by 141
  5. 2020Explainable Reinforcement Learning: A Surveycited by 141
  6. 2020Principles and Practice of Explainable Machine Learningcited by 99
  7. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  8. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  9. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  10. 2016Graying the black box: Understanding DQNscited by 68
  11. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  12. 2019Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanationscited by 64
  13. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  14. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  15. 2019Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attributioncited by 50
  16. 2018Establishing Appropriate Trust via Critical Statescited by 48
  17. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  18. 2020Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitionscited by 41
  19. 2018Transparency and Explanation in Deep Reinforcement Learning Neural Networkscited by 40
  20. 2019Generation of Policy-Level Explanations for Reinforcement Learningcited by 37
  21. 2017Enabling Robots to Communicate their Objectivescited by 35
  22. 2019Few-Shot Bayesian Imitation Learning with Logical Program Policiescited by 30
  23. 2020Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Mapscited by 30
  24. 2019Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policycited by 29
  25. 2018Towards Better Interpretability in Deep Q-Networkscited by 22
  26. 2020What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomescited by 20
  27. 2018Autonomous Self-Explanation of Behavior for Interactive Reinforcement Learning Agentscited by 10
  28. 2020Automatic Discovery of Interpretable Planning Strategiescited by 9
  29. 2020Self-Supervised Discovering of Interpretable Features for Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 9; 2025: 9; 2026: 9.

33 papers.

Sort:

By area:

  1. 2024Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Directioncited by 46
  2. 2023Formally Explaining Neural Networks within Reactive Systemscited by 24
  3. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  4. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  5. 2023Vision-based Learning for Drones: A Surveycited by 11
  6. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
  7. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8
  8. 2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Drivingcited by 7
  9. 2025Quantum computing and artificial intelligence: status and perspectivescited by 6
  10. 2023How to Control Hydrodynamic Force on Fluidic Pinball via Deep Reinforcement Learningcited by 5
  11. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  12. 2024Probabilistic Model Checking of Stochastic Reinforcement Learning Policiescited by 5
21 more
  1. 2024Interpretable end-to-end Neurosymbolic Reinforcement Learning agentscited by 4
  2. 2024ADESSE: Advice Explanations in Complex Repeated Decision-Making Environmentscited by 3
  3. 2024Explainable Post hoc Portfolio Management Financial Policy of a Deep Reinforcement Learning agentcited by 3
  4. 2024XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniquescited by 2
  5. 2025Co-Activation Graph Analysis of Safety-Verified and Explainable Deep Reinforcement Learning Policiescited by 2
  6. 2025Behaviour Discovery and Attribution for Explainable Reinforcement Learningcited by 2
  7. 2023Understanding User Preferences in Explainable Artificial Intelligence: A Survey and a Mapping Function Proposalcited by 1
  8. 2025Differentiable Weightless Controllers: Learning Logic Circuits for Continuous Controlcited by 1
  9. 2026BXRL: Behavior-Explainable Reinforcement Learningcited by 1
  10. 2024Unifying Interpretability and Explainability for Alzheimer's Disease Progression Predictioncited by 0
  11. 2025Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learningcited by 0
  12. 2025Algorithm-Relative Trajectory Valuation in Policy Gradient Controlcited by 0
  13. 2025Deep Learning Perspective of Scene Understanding in Autonomous Robotscited by 0
  14. 2026BRAIN: Bayesian Reasoning via Active Inference for Agentic and Embodied Intelligence in Mobile Networkscited by 0
  15. 2026COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenancecited by 0
  16. 2026TREX: Trajectory Explanations for Multi-Objective Reinforcement Learningcited by 0
  17. 2026Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learningcited by 0
  18. 2026GRAIL: Autonomous Concept Grounding for Neuro-Symbolic Reinforcement Learningcited by 0
  19. 2026Sample-efficient Neuro-symbolic Proximal Policy Optimizationcited by 0
  20. 2026Interpretable experiential learning based on state history and global feedbackcited by 0
  21. 2026Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedbackcited by 0