# Reinforcement Learning with Attention that Works: A Self-Supervised Approach

arXiv: https://arxiv.org/abs/1904.03367 · HTML: https://mlsanity.com/paper/1904.03367/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2019.

## Built on by (most cited first, 19 of 19)

- [Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control](https://mlsanity.com/paper/2103.00375/) (2021)
- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [Hindsight PRIORs for Reward Learning from Human Preferences](https://mlsanity.com/paper/2404.08828/) (2024)
- [Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.04067/) (2021)
- [MEDUSA: Multi-scale Encoder-Decoder Self-Attention Deep Neural Network Architecture for Medical Image Analysis](https://mlsanity.com/paper/2110.06063/) (2021)
- [Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning](https://mlsanity.com/paper/2112.03020/) (2021)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)
- [Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning](https://mlsanity.com/paper/2310.00229/) (2023)
- [Nearest-Neighbor-based Collision Avoidance for Quadrotors via Reinforcement Learning](https://mlsanity.com/paper/2104.14912/) (2021)
- [Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning](https://mlsanity.com/paper/2309.01458/) (2023)
- [Pretrained Encoders are All You Need](https://mlsanity.com/paper/2106.05139/) (2021)
- [BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning](https://mlsanity.com/paper/2403.01483/) (2024)
- [Deep Reinforcement Learning via Object-Centric Attention](https://mlsanity.com/paper/2504.03024/) (2025)
- [Goal-Conditioned Q-Learning as Knowledge Distillation](https://mlsanity.com/paper/2208.13298/) (2022)
- [Revealing Human Attention Patterns from Gameplay Analysis for Reinforcement Learning](https://mlsanity.com/paper/2504.11118/) (2025)
- [Benchmarking Deep Reinforcement Learning Algorithms for Vision-based Robotics](https://mlsanity.com/paper/2201.04224/) (2022)
- [Action Q-Transformer: Visual Explanation in Deep Reinforcement Learning with Encoder-Decoder Model using Action Query](https://mlsanity.com/paper/2306.13879/) (2023)
- [Pay Attention to What and Where? Interpretable Feature Extractor in Vision-based Deep Reinforcement Learning](https://mlsanity.com/paper/2504.10071/) (2025)
- [Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion](https://mlsanity.com/paper/2606.18469/) (2026)

## Abstract (arXiv)

Attention models have had a significant positive impact on deep learning across a range of tasks. However previous attempts at integrating attention with reinforcement learning have failed to produce significant improvements. We propose the first combination of self attention and reinforcement learning that is capable of producing significant improvements, including new state of the art results in the Arcade Learning Environment. Unlike the selective attention models used in previous attempts, which constrain the attention via preconceived notions of importance, our implementation utilises the Markovian properties inherent in the state input. Our method produces a faithful visualisation of the policy, focusing on the behaviour of the agent. Our experiments demonstrate that the trained policies use multiple simultaneous foci of attention, and are able to modulate attention over time to deal with situations of partial observability.