# Transparency and Explanation in Deep Reinforcement Learning Neural Networks

arXiv: https://arxiv.org/abs/1809.06061 · HTML: https://mlsanity.com/paper/1809.06061/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2018.

## Built on by (most cited first, 30 of 40)

- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approach](https://mlsanity.com/paper/2304.08349/) (2023)
- [Deep Interpretable Models of Theory of Mind](https://mlsanity.com/paper/2104.02938/) (2021)
- [A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning](https://mlsanity.com/paper/2505.19281/) (2025)
- [Explainable Artificial Intelligence (XAI) for Internet of Things: A Survey](https://mlsanity.com/paper/2206.04800/) (2022)
- [A Closer Look at Reward Decomposition for High-Level Robotic Explanations](https://mlsanity.com/paper/2304.12958/) (2023)
- [Explaining RL Decisions with Trajectories](https://mlsanity.com/paper/2305.04073/) (2023)
- [Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning](https://mlsanity.com/paper/2112.03020/) (2021)
- [Interpretable Learned Emergent Communication for Human-Agent Teams](https://mlsanity.com/paper/2201.07452/) (2022)
- [Know Your Model (KYM): Increasing Trust in AI and Machine Learning](https://mlsanity.com/paper/2106.11036/) (2021)
- [ProtoX: Explaining a Reinforcement Learning Agent via Prototyping](https://mlsanity.com/paper/2211.03162/) (2022)
- [Sufficient reasons for classifier decisions in the presence of constraints](https://mlsanity.com/paper/2105.06001/) (2021)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanations](https://mlsanity.com/paper/2109.13978/) (2021)
- [Policy Distillation with Selective Input Gradient Regularization for Efficient Interpretability](https://mlsanity.com/paper/2205.08685/) (2022)
- [Counterfactual Explanation Policies in RL](https://mlsanity.com/paper/2307.13192/) (2023)
- [Causal State Distillation for Explainable Reinforcement Learning](https://mlsanity.com/paper/2401.00104/) (2023)
- [XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniques](https://mlsanity.com/paper/2402.12685/) (2024)
- [Behaviour Discovery and Attribution for Explainable Reinforcement Learning](https://mlsanity.com/paper/2503.14973/) (2025)
- [Explainability-aided Domain Generalization for Image Classification](https://mlsanity.com/paper/2104.01742/) (2021)
- [Trajectory-Constrained Deep Latent Visual Attention for Improved Local Planning in Presence of Heterogeneous Terrain](https://mlsanity.com/paper/2112.04684/) (2021)
- [Power and accountability in reinforcement learning applications to environmental policy](https://mlsanity.com/paper/2205.10911/) (2022)
- [Causal Explanation for Reinforcement Learning: Quantifying State and Temporal Importance](https://mlsanity.com/paper/2210.13507/) (2022)
- [XAI-CF -- Examining the Role of Explainable Artificial Intelligence in Cyber Forensics](https://mlsanity.com/paper/2402.02452/) (2024)
- ['Explaining RL Decisions with Trajectories': A Reproducibility Study](https://mlsanity.com/paper/2411.07200/) (2024)
- [Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning](https://mlsanity.com/paper/2411.16120/) (2024)

## Abstract (arXiv)

Autonomous AI systems will be entering human society in the near future to provide services and work alongside humans. For those systems to be accepted and trusted, the users should be able to understand the reasoning process of the system, i.e. the system should be transparent. System transparency enables humans to form coherent explanations of the system's decisions and actions. Transparency is important not only for user trust, but also for software debugging and certification. In recent years, Deep Neural Networks have made great advances in multiple application areas. However, deep neural networks are opaque. In this paper, we report on work in transparency in Deep Reinforcement Learning Networks (DRLN). Such networks have been extremely successful in accurately learning action control in image input domains, such as Atari games. In this paper, we propose a novel and general method that (a) incorporates explicit object recognition processing into deep reinforcement learning models, (b) forms the basis for the development of "object saliency maps", to provide visualization of internal states of DRLNs, thus enabling the formation of explanations and (c) can be incorporated in any existing deep reinforcement learning framework. We present computational results and human experiments to evaluate our approach.