mlsanityData to 14 Oct 2026

Fast and Data-Efficient Training of Rainbow: an Experimental Study on Atari

arXiv 2111.10247 ↗First posted 19 Nov 2021cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Across the Arcade Learning Environment, Rainbow achieves a level of performance competitive with humans and modern RL algorithms. However, attaining this level of performance requires large amounts of data and hardware resources, making research in this area computationally expensive and use in practical applications often infeasible. This paper's contribution is threefold: We (1) propose an improved version of Rainbow, seeking to drastically reduce Rainbow's data, training time, and compute requirements while maintaining its competitive performance; (2) we empirically demonstrate the effectiveness of our approach through experiments on the Arcade Learning Environment, and (3) we conduct a number of ablation studies to investigate the effect of the individual proposed modifications. Our improved version of Rainbow reaches a median human normalized score close to classic Rainbow's, while using 20 times less data and requiring only 7.5 hours of training time on a single GPU. We also provide our full implementation including pre-trained models.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 51 of its 74 references that are in the map. First posted on arXiv 19 Nov 2021.

Builds on

51 of 74 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  12. 2017Mixed Precision Trainingcited by 1,156
38 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  2. 2015Prioritized Experience Replaycited by 1,099
  3. 2020Mastering Atari with Discrete World Modelscited by 1,007
  4. 2018Exploration by Random Network Distillationcited by 993
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  12. 2019Solving Rubik's Cube with a Robot Handcited by 734
  13. 2018Learning Dexterous In-Hand Manipulationcited by 687
  14. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  15. 2019Model-Based Reinforcement Learning for Ataricited by 502
  16. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  17. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  18. 2020Reinforcement Learning with Augmented Datacited by 468
  19. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  20. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 370
  21. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  22. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  23. 2019Challenges of Real-World Reinforcement Learningcited by 291
  24. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  25. 2018Distributed Prioritized Experience Replaycited by 269
  26. 2017Noisy Networks for Explorationcited by 266
  27. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  28. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  29. 2019When to use parametric models in reinforcement learning?cited by 118
  30. 2020Chip Placement with Deep Reinforcement Learningcited by 83
  31. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  32. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  33. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  34. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  35. 2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
  36. 2020D2RL: Deep Dense Architectures in Reinforcement Learningcited by 42
  37. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  38. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 4.

10 papers.

  1. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  2. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  3. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  4. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  5. 2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
  6. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  7. 2025Contextual Similarity Distillation: Ensemble Uncertainties with a Single Modelcited by 3
  8. 2025Pay Attention to What and Where? Interpretable Feature Extractor in Vision-based Deep Reinforcement Learningcited by 0
  9. 2025DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulationcited by 0
  10. 2025Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slowcited by 0