mlsanityData to 14 Oct 2026

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC

arXiv 2411.03820 ↗First posted 6 Nov 2024cs.AI cs.LGcited by 7

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent's performance. In this paper, we present "Beyond The Rainbow" (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC, with a human-normalized interquartile mean (IQM) of 7.4 on Atari-60. Beyond Atari, we demonstrate BTR's capability to handle complex 3D games, successfully training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algorithmic changes. Designing BTR with computational efficiency in mind, agents can be trained using a high-end desktop PC on 200 million Atari frames within 12 hours. Additionally, we conduct detailed ablation studies of each component, analyzing the performance and impact using numerous measures. Code is available at https://github.com/VIPTankz/BTR.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 58 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

46 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale13 Oct
  2. 2024Simplifying Deep Temporal Difference Learning5 Jul
  3. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning25 Jun
  4. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data1 Mar
  5. 2024Disentangling the Causes of Plasticity Loss in Neural Networks29 Feb

Other references · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2015Prioritized Experience Replaycited by 1,099
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  11. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 765
29 more
  1. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  2. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  4. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  5. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  6. 2018Distributed Prioritized Experience Replaycited by 269
  7. 2017Noisy Networks for Explorationcited by 266
  8. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  9. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  10. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  11. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  12. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  13. 2020Phasic Policy Gradientcited by 123
  14. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  15. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  16. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 81
  17. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  18. 2020Munchausen Reinforcement Learningcited by 69
  19. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  20. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  21. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  22. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  23. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  24. 2023Small batch deep reinforcement learningcited by 36
  25. 2022The Phenomenon of Policy Churncited by 28
  26. 2022Human-level Atari 200x fastercited by 28
  27. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  28. 2019Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing fieldcited by 15
  29. 2021Fast and Data-Efficient Training of Rainbow: an Experimental Study on Ataricited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Toward Agents That Reason About Their Computationcited by 3
  2. 2025A Comparative User Evaluation of XRL Explanations using Goal Identificationcited by 1
  3. 2025Unraveling the Rainbow: can value-based methods schedule?cited by 0
  4. 2025Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slowcited by 0
  5. 2026Distributional Value Estimation Without Target Networks for Robust Quality-Diversitycited by 0
  6. 2026Scalable Reinforcement Learning via Adaptive Batch Scalingcited by 0
  7. 2026Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Transforming Game Play: A Comparative Study of DCQN and DTQN Architectures in Reinforcement Learning
  2. 2024FDQN: A Flexible Deep Q-Network Framework for Game Automation
  3. 2024A Comparative Study of Deep Reinforcement Learning Models: DQN vs PPO vs A2C
  4. 2024PG-Rainbow: Using Distributional Reinforcement Learning in Policy Gradient Methods
  5. 2025Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations
  6. 2025Deep Reinforcement Learning in Applied Control: Challenges, Analysis, and Insights
  7. 2025Automatic Reward Shaping from Confounded Offline Data
  8. 2025Deep Double Q-learning