# State of the Art Control of Atari Games Using Shallow Reinforcement Learning

arXiv: https://arxiv.org/abs/1512.01563 · HTML: https://mlsanity.com/paper/1512.01563/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2015.

## Built on by (most cited first, 23 of 23)

- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Investigating the Properties of Neural Network Representations in Reinforcement Learning](https://mlsanity.com/paper/2203.15955/) (2022)
- [Structured World Belief for Reinforcement Learning in POMDP](https://mlsanity.com/paper/2107.08577/) (2021)
- [On the Convergence of SARSA with Linear Function Approximation](https://mlsanity.com/paper/2202.06828/) (2022)
- [Harnessing Discrete Representations For Continual Reinforcement Learning](https://mlsanity.com/paper/2312.01203/) (2023)
- [Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search](https://mlsanity.com/paper/2212.14849/) (2022)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control](https://mlsanity.com/paper/2210.11262/) (2022)
- [Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features](https://mlsanity.com/paper/2409.12135/) (2024)
- [Agent-State Construction with Auxiliary Inputs](https://mlsanity.com/paper/2211.07805/) (2022)
- [Fast and Data Efficient Reinforcement Learning from Pixels via Non-Parametric Value Approximation](https://mlsanity.com/paper/2203.03078/) (2022)
- [Learning Goal-Conditioned Representations for Language Reward Models](https://mlsanity.com/paper/2407.13887/) (2024)
- [Performance Variation in Deep Reinforcement Learning](https://mlsanity.com/paper/2606.06746/) (2026)
- [On the impact of MDP design for Reinforcement Learning agents in Resource Management](https://mlsanity.com/paper/2109.03202/) (2021)
- [Is Policy Learning Overrated?: Width-Based Planning and Active Learning for Atari](https://mlsanity.com/paper/2109.15310/) (2021)
- [Resource Governance in Networked Systems via Integrated Variational Autoencoders and Reinforcement Learning](https://mlsanity.com/paper/2410.23393/) (2024)
- [Width-based Lookaheads with Learnt Base Policies and Heuristics Over the Atari-2600 Benchmark](https://mlsanity.com/paper/2106.12151/) (2021)
- [Improving Experience Replay through Modeling of Similar Transitions' Sets](https://mlsanity.com/paper/2111.06907/) (2021)
- [Social Interpretable Reinforcement Learning](https://mlsanity.com/paper/2401.15480/) (2024)
- [Neuroevolution of Self-Attention Over Proto-Objects](https://mlsanity.com/paper/2505.00186/) (2025)
- [Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer](https://mlsanity.com/paper/2509.24947/) (2025)

## Abstract (arXiv)

The recently introduced Deep Q-Networks (DQN) algorithm has gained attention as one of the first successful combinations of deep neural networks and reinforcement learning. Its promise was demonstrated in the Arcade Learning Environment (ALE), a challenging framework composed of dozens of Atari 2600 games used to evaluate general competency in AI. It achieved dramatically better results than earlier approaches, showing that its ability to learn good representations is quite robust and general. This paper attempts to understand the principles that underlie DQN's impressive performance and to better contextualize its success. We systematically evaluate the importance of key representational biases encoded by DQN's network by proposing simple linear representations that make use of these concepts. Incorporating these characteristics, we obtain a computationally practical feature set that achieves competitive performance to DQN in the ALE. Besides offering insight into the strengths and weaknesses of DQN, we provide a generic representation for the ALE, significantly reducing the burden of learning a representation for each game. Moreover, we also provide a simple, reproducible benchmark for the sake of comparison to future work in the ALE.