# Bigger, Better, Faster: Human-level Atari with human-level efficiency

arXiv: https://arxiv.org/abs/2305.19452 · HTML: https://mlsanity.com/paper/2305.19452/

## Key facts

- Cited by 147 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 72 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 30 of 50)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)

## Built on by (most cited first, 30 of 147)

- [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024)
- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control](https://mlsanity.com/paper/2405.16158/) (2024)
- [STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning](https://mlsanity.com/paper/2310.09615/) (2023)
- [Mixtures of Experts Unlock Parameter Scaling for Deep RL](https://mlsanity.com/paper/2402.08609/) (2024)
- [SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning](https://mlsanity.com/paper/2410.09754/) (2024)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning](https://mlsanity.com/paper/2403.00514/) (2024)
- [DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization](https://mlsanity.com/paper/2310.19668/) (2023)
- [1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities](https://mlsanity.com/paper/2503.14858/) (2025)
- [FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control](https://mlsanity.com/paper/2505.22642/) (2025)
- [Maintaining Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2306.13812/) (2023)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [Towards General-Purpose Model-Free Reinforcement Learning](https://mlsanity.com/paper/2501.16142/) (2025)
- [EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data](https://mlsanity.com/paper/2403.00564/) (2024)
- [In value-based deep reinforcement learning, a pruned network is a good network](https://mlsanity.com/paper/2402.12479/) (2024)
- [Hyperspherical Normalization for Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2502.15280/) (2025)
- [Small batch deep reinforcement learning](https://mlsanity.com/paper/2310.03882/) (2023)
- [EXPO: Stable Reinforcement Learning with Expressive Policies](https://mlsanity.com/paper/2507.07986/) (2025)
- [Efficient World Models with Context-Aware Tokenization](https://mlsanity.com/paper/2406.19320/) (2024)
- [Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages](https://mlsanity.com/paper/2310.07418/) (2023)
- [A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning](https://mlsanity.com/paper/2405.19153/) (2024)
- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [Plasticity Loss in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2411.04832/) (2024)
- [Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners](https://mlsanity.com/paper/2505.23150/) (2025)
- [Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning](https://mlsanity.com/paper/2506.15544/) (2025)
- [Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn](https://mlsanity.com/paper/2506.00592/) (2025)
- [On the consistency of hyper-parameter selection in value-based deep reinforcement learning](https://mlsanity.com/paper/2406.17523/) (2024)
- [Mixture of Experts in a Mixture of RL settings](https://mlsanity.com/paper/2406.18420/) (2024)
- [Retrieval-Augmented Decision Transformer: External Memory for In-context RL](https://mlsanity.com/paper/2410.07071/) (2024)

## Abstract (arXiv)

We introduce a value-based RL agent, which we call BBF, that achieves super-human performance in the Atari 100K benchmark. BBF relies on scaling the neural networks used for value estimation, as well as a number of other design choices that enable this scaling in a sample-efficient manner. We conduct extensive analyses of these design choices and provide insights for future work. We end with a discussion about updating the goalposts for sample-efficient RL research on the ALE. We make our code and data publicly available at https://github.com/google-research/google-research/tree/master/bigger_better_faster.