# Towards Interpretable Reinforcement Learning Using Attention Augmented Agents

arXiv: https://arxiv.org/abs/1906.02500 · HTML: https://mlsanity.com/paper/1906.02500/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2019.

## Built on by (most cited first, 30 of 82)

- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning](https://mlsanity.com/paper/2101.12446/) (2021)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Deep Transformer Q-Networks for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2206.01078/) (2022)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control](https://mlsanity.com/paper/2103.00375/) (2021)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Self-Interpretable Model with TransformationEquivariant Interpretation](https://mlsanity.com/paper/2111.04927/) (2021)
- [Exploring the Promise and Limits of Real-Time Recurrent Learning](https://mlsanity.com/paper/2305.19044/) (2023)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [A Survey of State Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17518/) (2025)
- [Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspective](https://mlsanity.com/paper/2404.06492/) (2024)
- [Learning to Generalize with Object-centric Agents in the Open World Survival Game Crafter](https://mlsanity.com/paper/2208.03374/) (2022)
- [A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning](https://mlsanity.com/paper/2505.19281/) (2025)
- [Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.04067/) (2021)
- [Active Sensing for Communications by Learning](https://mlsanity.com/paper/2112.04075/) (2021)
- [Learning to Identify Critical States for Reinforcement Learning from Videos](https://mlsanity.com/paper/2308.07795/) (2023)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Joint Attention for Multi-Agent Coordination and Social Learning](https://mlsanity.com/paper/2104.07750/) (2021)
- [Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning](https://mlsanity.com/paper/2110.12985/) (2021)
- [Agent-Centric Representations for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2104.09402/) (2021)
- [Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning](https://mlsanity.com/paper/2112.03020/) (2021)
- [Toward Causal-Aware RL: State-Wise Action-Refined Temporal Difference](https://mlsanity.com/paper/2201.00354/) (2022)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)

## Abstract (arXiv)

Inspired by recent work in attention models for image captioning and question answering, we present a soft attention model for the reinforcement learning domain. This model uses a soft, top-down attention mechanism to create a bottleneck in the agent, forcing it to focus on task-relevant information by sequentially querying its view of the environment. The output of the attention mechanism allows direct observation of the information used by the agent to select its actions, enabling easier interpretation of this model than of traditional models. We analyze different strategies that the agents learn and show that a handful of strategies arise repeatedly across different games. We also show that the model learns to query separately about space and content (`where' vs. `what'). We demonstrate that an agent using this mechanism can achieve performance competitive with state-of-the-art models on ATARI tasks while still being interpretable.