# Towards Better Interpretability in Deep Q-Networks

arXiv: https://arxiv.org/abs/1809.05630 · HTML: https://mlsanity.com/paper/1809.05630/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2018.

## Built on by (most cited first, 22 of 22)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data](https://mlsanity.com/paper/2306.03346/) (2023)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Deep Interpretable Models of Theory of Mind](https://mlsanity.com/paper/2104.02938/) (2021)
- [Time-Efficient Reward Learning via Visually Assisted Cluster Ranking](https://mlsanity.com/paper/2212.00169/) (2022)
- [A Closer Look at Reward Decomposition for High-Level Robotic Explanations](https://mlsanity.com/paper/2304.12958/) (2023)
- [Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning](https://mlsanity.com/paper/2112.03020/) (2021)
- [ProtoX: Explaining a Reinforcement Learning Agent via Prototyping](https://mlsanity.com/paper/2211.03162/) (2022)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning](https://mlsanity.com/paper/2309.01458/) (2023)
- [FLARE: Fingerprinting Deep Reinforcement Learning Agents using Universal Adversarial Masks](https://mlsanity.com/paper/2307.14751/) (2023)
- [Learning Goal-Conditioned Representations for Language Reward Models](https://mlsanity.com/paper/2407.13887/) (2024)
- [An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention](https://mlsanity.com/paper/2109.08360/) (2021)
- [XDQN: Inherently Interpretable DQN through Mimicking](https://mlsanity.com/paper/2301.03043/) (2023)
- [Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning](https://mlsanity.com/paper/2411.16120/) (2024)
- [Interpretable Option Discovery using Deep Q-Learning and Variational Autoencoders](https://mlsanity.com/paper/2210.01231/) (2022)
- [Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learning](https://mlsanity.com/paper/2511.20591/) (2025)

## Abstract (arXiv)

Deep reinforcement learning techniques have demonstrated superior performance in a wide variety of environments. As improvements in training algorithms continue at a brisk pace, theoretical or empirical studies on understanding what these networks seem to learn, are far behind. In this paper we propose an interpretable neural network architecture for Q-learning which provides a global explanation of the model's behavior using key-value memories, attention and reconstructible embeddings. With a directed exploration strategy, our model can reach training rewards comparable to the state-of-the-art deep Q-learning models. However, results suggest that the features extracted by the neural network are extremely shallow and subsequent testing using out-of-sample examples shows that the agent can easily overfit to trajectories seen during training.