# Observe and Look Further: Achieving Consistent Performance on Atari

arXiv: https://arxiv.org/abs/1805.11593 · HTML: https://mlsanity.com/paper/1805.11593/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2018.

## Built on by (most cited first, 30 of 44)

- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [State Alignment-based Imitation Learning](https://mlsanity.com/paper/1911.10947/) (2019)
- [Offline Learning from Demonstrations and Unlabeled Experience](https://mlsanity.com/paper/2011.13885/) (2020)
- [Robust Scheduling with GFlowNets](https://mlsanity.com/paper/2302.05446/) (2023)
- [DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization](https://mlsanity.com/paper/2112.04716/) (2021)
- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Task-Relevant Adversarial Imitation Learning](https://mlsanity.com/paper/1910.01077/) (2019)
- [On Bonus-Based Exploration Methods in the Arcade Learning Environment](https://mlsanity.com/paper/2109.11052/) (2021)
- [Regularized Behavior Value Estimation](https://mlsanity.com/paper/2103.09575/) (2021)
- [Human-level Atari 200x faster](https://mlsanity.com/paper/2209.07550/) (2022)
- [Return-based Scaling: Yet Another Normalisation Trick for Deep RL](https://mlsanity.com/paper/2105.05347/) (2021)
- [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://mlsanity.com/paper/2310.08348/) (2023)
- [Scaling data-driven robotics with reward sketching and batch reinforcement learning](https://mlsanity.com/paper/1909.12200/) (2019)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning](https://mlsanity.com/paper/2110.14118/) (2021)
- [Reward Centering](https://mlsanity.com/paper/2405.09999/) (2024)
- [Learning Dynamics and Generalization in Reinforcement Learning](https://mlsanity.com/paper/2206.02126/) (2022)
- [Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation](https://mlsanity.com/paper/2112.00597/) (2021)
- [Visualizing MuZero Models](https://mlsanity.com/paper/2102.12924/) (2021)
- [Convergent and Efficient Deep Q Network Algorithm](https://mlsanity.com/paper/2106.15419/) (2021)
- [Efficient Offline Policy Optimization with a Learned Model](https://mlsanity.com/paper/2210.05980/) (2022)
- [Self-Consistent Models and Values](https://mlsanity.com/paper/2110.12840/) (2021)
- [ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/1910.12154/) (2019)
- [Taylor Expansion of Discount Factors](https://mlsanity.com/paper/2106.06170/) (2021)
- [Enhancing Reinforcement Learning Agents with Local Guides](https://mlsanity.com/paper/2402.13930/) (2024)
- [Spectral Bellman Method: Unifying Representation and Exploration in RL](https://mlsanity.com/paper/2507.13181/) (2025)
- [Adapting to Reward Progressivity via Spectral Reinforcement Learning](https://mlsanity.com/paper/2104.14138/) (2021)
- [Policy-Based Self-Competition for Planning Problems](https://mlsanity.com/paper/2306.04403/) (2023)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Conjugated Discrete Distributions for Distributional Reinforcement Learning](https://mlsanity.com/paper/2112.07424/) (2021)

## Abstract (arXiv)

Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games: processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently. In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games. A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of γ = 0.999 (instead of γ = 0.99) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states. When tested on a set of 42 Atari games, our algorithm exceeds the performance of an average human on 40 games using a common set of hyper parameters. Furthermore, it is the first deep RL algorithm to solve the first level of Montezuma's Revenge.