mlsanityData to 7 Oct 2026

Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

arXiv 2011.14826 ↗First posted 20 Nov 2020cs.LG cs.AIcited by 78

Not on the map. Why?

Abstract (arXiv)

Since the introduction of DQN, a vast majority of reinforcement learning research has focused on reinforcement learning with deep neural networks as function approximators. New methods are typically evaluated on a set of environments that have now become standard, such as Atari 2600 games. While these benchmarks help standardize evaluation, their computational cost has the unfortunate side effect of widening the gap between those with ample access to computational resources, and those without. In this work we argue that, despite the community's emphasis on large-scale environments, the traditional small-scale environments can still yield valuable scientific insights and can help reduce the barriers to entry for underprivileged communities. To substantiate our claims, we empirically revisit the paper which introduced the Rainbow algorithm [Hessel et al., 2018] and present some new insights into the algorithms used by Rainbow.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 17; 2023: 15; 2024: 14; 2025: 11; 2026: 6.

78 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  3. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  4. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  5. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  6. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  7. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  8. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  9. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  10. 2023Empirical Design in Reinforcement Learningcited by 54
  11. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  12. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 45
38 more
  1. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  2. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  3. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  4. 2023Small batch deep reinforcement learningcited by 35
  5. 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
  6. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  7. 2021Adaptive Rational Activations to Boost Deep Reinforcement Learningcited by 23
  8. 2021On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learningcited by 23
  9. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  10. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  11. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  12. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  13. 2022Proper Reuse of Image Classification Features Improves Object Detectioncited by 20
  14. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  15. 2023PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clippingcited by 17
  16. 2021Large Batch Experience Replaycited by 14
  17. 2022Robots Enact Malignant Stereotypescited by 14
  18. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  19. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12
  20. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  21. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  22. 2021Fast and Data-Efficient Training of Rainbow: an Experimental Study on Ataricited by 10
  23. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  24. 2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
  25. 2022Robust Losses for Learning Value Functionscited by 7
  26. 2024Neural-Kernel Conditional Mean Embeddingscited by 7
  27. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  28. 2022No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RLcited by 6
  29. 2022CUP: Critic-Guided Policy Reusecited by 6
  30. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  31. 2021ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectivescited by 5
  32. 2022Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworkscited by 5
  33. 2025Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAXcited by 5
  34. 2021When does loss-based prioritization fail?cited by 4
  35. 2021Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspectivecited by 4
  36. 2022Reinforcement Learning for Economic Policy: A New Frontier?cited by 4
  37. 2022How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?cited by 4
  38. 2022MSRL: Distributed Reinforcement Learning with Dataflow Fragmentscited by 4

50 most cited of 78 All 78 (JSON)