mlsanityData to 7 Oct 2026

Human-level Atari 200x faster

arXiv 2209.07550 ↗First posted 15 Sep 2022cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

The task of building general agents that perform well over a wide range of tasks has been an important goal in reinforcement learning since its inception. The problem has been subject of research of a large body of work, with performance frequently measured by observing scores over the wide range of environments contained in the Atari 57 benchmark. Agent57 was the first agent to surpass the human benchmark on all 57 games, but this came at the cost of poor data-efficiency, requiring nearly 80 billion frames of experience to achieve. Taking Agent57 as a starting point, we employ a diverse set of strategies to achieve a 200-fold reduction of experience needed to out perform the human baseline. We investigate a range of instabilities and bottlenecks we encountered while reducing the data regime, and propose effective solutions to build a more robust and efficient agent. We also demonstrate competitive performance with high-performing methods such as Muesli and MuZero. The four key components to our approach are (1) an approximate trust region method which enables stable bootstrapping from the online network, (2) a normalisation scheme for the loss and priorities which improves robustness when learning a set of value functions with a wide range of scales, (3) an improved architecture employing techniques from NFNets in order to leverage deeper networks without the need for normalization layers, and (4) a policy distillation method which serves to smooth out the instantaneous greedy policy overtime.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 39 of its 66 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

39 of 66 references.

References · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2016Deep Networks with Stochastic Depthcited by 973
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
27 more
  1. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  2. 2016Deep Exploration via Bootstrapped DQNcited by 518
  3. 2019Model-Based Reinforcement Learning for Ataricited by 495
  4. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  5. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  6. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  7. 2019Challenges of Real-World Reinforcement Learningcited by 282
  8. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  9. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  10. 2018Distributed Prioritized Experience Replaycited by 260
  11. 2020First return, then explorecited by 256
  12. 2021Mastering Atari Games with Limited Datacited by 235
  13. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  14. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  15. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  16. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  17. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  18. 2019When to use parametric models in reinforcement learning?cited by 118
  19. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  20. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  21. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  22. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  23. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  24. 2022The Phenomenon of Policy Churncited by 28
  25. 2021Reverb: A Framework For Experience Replaycited by 27
  26. 2021Return-based Scaling: Yet Another Normalisation Trick for Deep RLcited by 22
  27. 2022Fast and Data Efficient Reinforcement Learning from Pixels via Non-Parametric Value Approximationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 6; 2025: 7; 2026: 3.

28 papers.

Sort:

By area:

  1. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  2. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  3. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  4. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  5. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  6. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  7. 2023Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selectioncited by 14
  8. 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
  9. 2023Sample Efficient Deep Reinforcement Learning via Local Planningcited by 9
  10. 2023Unlocking the Power of Representations in Long-term Novelty-based Explorationcited by 8
  11. 2023Bridging the Sim-to-Real Gap from the Information Bottleneck Perspectivecited by 8
  12. 2023Reinforcement Learning with Fast and Forgetful Memorycited by 8
16 more
  1. 2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
  2. 2024Robot Policy Learning with Temporal Optimal Transport Rewardcited by 8
  3. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  4. 2023An Invitation to Deep Reinforcement Learningcited by 5
  5. 2025Simple, Good, Fast: Self-Supervised World Models Free of Baggagecited by 3
  6. 2025Generalization in Reinforcement Learning for Radio Access Networkscited by 2
  7. 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learningcited by 2
  8. 2026Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learningcited by 1
  9. 2023Action Q-Transformer: Visual Explanation in Deep Reinforcement Learning with Encoder-Decoder Model using Action Querycited by 0
  10. 2024Learning in complex action spaces without policy gradientscited by 0
  11. 2025Pay Attention to What and Where? Interpretable Feature Extractor in Vision-based Deep Reinforcement Learningcited by 0
  12. 2025SPLASH! Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrationscited by 0
  13. 2025DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulationcited by 0
  14. 2025Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slowcited by 0
  15. 2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Explorationcited by 0
  16. 2026Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discoverycited by 0