# Visualizing and Understanding Atari Agents

arXiv: https://arxiv.org/abs/1711.00138 · HTML: https://mlsanity.com/paper/1711.00138/

## Key facts

- Cited by 141 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2017.

## Built on by (most cited first, 30 of 141)

- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies](https://mlsanity.com/paper/2106.09678/) (2021)
- [Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning](https://mlsanity.com/paper/2101.12446/) (2021)
- [Counterfactual Explanations in Sequential Decision Making Under Uncertainty](https://mlsanity.com/paper/2107.02776/) (2021)
- [Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems](https://mlsanity.com/paper/2109.09904/) (2021)
- [Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation](https://mlsanity.com/paper/2102.05714/) (2021)
- [Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning](https://mlsanity.com/paper/2209.09203/) (2022)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values](https://mlsanity.com/paper/2110.01307/) (2021)
- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [Quantization-Aware Imitation-Learning for Resource-Efficient Robotic Control](https://mlsanity.com/paper/2412.01034/) (2024)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Investigating the Properties of Neural Network Representations in Reinforcement Learning](https://mlsanity.com/paper/2203.15955/) (2022)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [ExoMiner: A Highly Accurate and Explainable Deep Learning Classifier that Validates 301 New Exoplanets](https://mlsanity.com/paper/2111.10009/) (2021)
- [End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations](https://mlsanity.com/paper/2403.12451/) (2024)
- [A Survey of State Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17518/) (2025)
- [A Game-Theoretic Taxonomy of Visual Concepts in DNNs](https://mlsanity.com/paper/2106.10938/) (2021)
- [Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes](https://mlsanity.com/paper/2312.11118/) (2023)
- [Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2210.11846/) (2022)
- [Critic Guided Segmentation of Rewarding Objects in First-Person Views](https://mlsanity.com/paper/2107.09540/) (2021)
- [Hindsight PRIORs for Reward Learning from Human Preferences](https://mlsanity.com/paper/2404.08828/) (2024)
- [Saliency-Aware Quantized Imitation Learning for Efficient Robotic Control](https://mlsanity.com/paper/2505.15304/) (2025)

## Abstract (arXiv)

While deep reinforcement learning (deep RL) agents are effective at maximizing rewards, it is often unclear what strategies they use to do so. In this paper, we take a step toward explaining deep RL agents through a case study using Atari 2600 environments. In particular, we focus on using saliency maps to understand how an agent learns and executes a policy. We introduce a method for generating useful saliency maps and use it to show 1) what strong agents attend to, 2) whether agents are making decisions for the right or wrong reasons, and 3) how agents evolve during learning. We also test our method on non-expert human subjects and find that it improves their ability to reason about these agents. Overall, our results show that saliency information can provide significant insight into an RL agent's decisions and learning behavior.