mlsanityData to 14 Oct 2026

Mastering Atari Games with Limited Data

arXiv 2111.00210 ↗First posted 30 Oct 2021cs.LG cs.AI cs.CVcited by 237

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at https://github.com/YeWR/EfficientZero. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.

Key facts. Cited by 237 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 30 Oct 2021.

Builds on

35 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May

Other references · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  12. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
20 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  2. 2015Prioritized Experience Replaycited by 1,099
  3. 2018DeepMind Control Suitecited by 956
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  9. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  10. 2019Model-Based Reinforcement Learning for Ataricited by 502
  11. 2020Reinforcement Learning with Augmented Datacited by 468
  12. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  13. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  14. 2019When to use parametric models in reinforcement learning?cited by 118
  15. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  16. 2021Learning and Planning in Complex Action Spacescited by 77
  17. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  18. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 54
  19. 2019ELF OpenGo: An Analysis and Open Reimplementation of AlphaZerocited by 39
  20. 2021Visualizing MuZero Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 38; 2023: 50; 2024: 56; 2025: 61; 2026: 31.

237 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,253
  2. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  3. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  5. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  6. 2022Transformers are Sample-Efficient World Modelscited by 328
  7. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  8. 2022Transformers Learn Shortcuts to Automatacited by 268
  9. 2023A Cookbook of Self-Supervised Learningcited by 250
  10. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  11. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  12. 2022Masked World Models for Visual Controlcited by 170
38 more
  1. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  2. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 139
  3. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  4. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 108
  5. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 89
  6. 2024World Models for Autonomous Driving: An Initial Surveycited by 80
  7. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  8. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  9. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 57
  10. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  11. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  12. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 45
  13. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  14. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  15. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  16. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 41
  17. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 40
  18. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  19. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 39
  20. 2024Mastering Memory Tasks with World Modelscited by 38
  21. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  22. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  23. 2024Efficient World Models with Context-Aware Tokenizationcited by 33
  24. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  25. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
  26. 2022Human-level Atari 200x fastercited by 28
  27. 2023MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulationcited by 28
  28. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoningcited by 28
  29. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 27
  30. 2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attackscited by 26
  31. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  32. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 26
  33. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 25
  34. 2023SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networkscited by 24
  35. 2024Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spacescited by 24
  36. 2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
  37. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  38. 2022Planning for Sample Efficient Imitation Learningcited by 20

50 most cited of 237 All 237 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
  2. 2024Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning
  3. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
  4. 2025Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations
  5. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
  6. 2025Performance Asymmetry in Model-Based Reinforcement Learning
  7. 2022Reliable validation of Reinforcement Learning Benchmarks
  8. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning