mlsanityData to 7 Oct 2026

Observe and Look Further: Achieving Consistent Performance on Atari

arXiv 1805.11593 ↗First posted 29 May 2018cs.LG cs.AI stat.MLcited by 42

Not on the map. Why?

Abstract (arXiv)

Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games: processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently. In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games. A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of γ = 0.999 (instead of γ = 0.99) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states. When tested on a set of 42 Atari games, our algorithm exceeds the performance of an average human on 40 games using a common set of hyper parameters. Furthermore, it is the first deep RL algorithm to solve the first level of Montezuma's Revenge.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2019: 4; 2020: 1; 2021: 16; 2022: 10; 2023: 6; 2024: 4; 2025: 1.

42 papers.

Sort:

By area:

  1. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  2. 2019State Alignment-based Imitation Learningcited by 73
  3. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  4. 2023Robust Scheduling with GFlowNetscited by 67
  5. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  6. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  7. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  8. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
  9. 2021Regularized Behavior Value Estimationcited by 32
  10. 2022Human-level Atari 200x fastercited by 28
  11. 2021Return-based Scaling: Yet Another Normalisation Trick for Deep RLcited by 22
  12. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
30 more
  1. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  2. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  3. 2021Object-Aware Regularization for Addressing Causal Confusion in Imitation Learningcited by 16
  4. 2024Reward Centeringcited by 16
  5. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13
  6. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  7. 2021Visualizing MuZero Modelscited by 9
  8. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8
  9. 2022Efficient Offline Policy Optimization with a Learned Modelcited by 8
  10. 2021Self-Consistent Models and Valuescited by 7
  11. 2019ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrationscited by 5
  12. 2021Taylor Expansion of Discount Factorscited by 4
  13. 2024Enhancing Reinforcement Learning Agents with Local Guidescited by 4
  14. 2023Policy-Based Self-Competition for Planning Problemscited by 3
  15. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  16. 2025Spectral Bellman Method: Unifying Representation and Exploration in RLcited by 3
  17. 2021Adapting to Reward Progressivity via Spectral Reinforcement Learningcited by 2
  18. 2021Conjugated Discrete Distributions for Distributional Reinforcement Learningcited by 2
  19. 2022MAN: Multi-Action Networks Learningcited by 2
  20. 2024ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyzecited by 2
  21. 2022Deep Q-learning: a robust control approachcited by 1
  22. 2022Mastering Spatial Graph Prediction of Road Networkscited by 1
  23. 2021DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learningcited by 0
  24. 2022On the benefits of non-linear weight updatescited by 0
  25. 2022Network Aware Compute and Memory Allocation in Optically Composable Data Centres with Deep Reinforcement Learning and Graph Neural Networkscited by 0
  26. 2022Generalization Through the Lens of Learning Dynamicscited by 0
  27. 2022Control of Continuous Quantum Systems with Many Degrees of Freedom based on Convergent Reinforcement Learningcited by 0
  28. 2023Plume: A Framework for High Performance Deep RL Network Controllers via Prioritized Trace Samplingcited by 0
  29. 2023CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systemscited by 0
  30. 2023Deep reinforcement learning uncovers processes for separating azeotropic mixtures without prior knowledgecited by 0