# Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

arXiv: https://arxiv.org/abs/2011.14826 · HTML: https://mlsanity.com/paper/2011.14826/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2020.

## Built on by (most cited first, 30 of 78)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Stop Regressing: Training Value Functions via Classification for Scalable Deep RL](https://mlsanity.com/paper/2403.03950/) (2024)
- [Mixtures of Experts Unlock Parameter Scaling for Deep RL](https://mlsanity.com/paper/2402.08609/) (2024)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective](https://mlsanity.com/paper/2105.05246/) (2021)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [Continual Learning with Foundation Models: An Empirical Study of Latent Replay](https://mlsanity.com/paper/2205.00329/) (2022)
- [Hyperparameters in Reinforcement Learning and How To Tune Them](https://mlsanity.com/paper/2306.01324/) (2023)
- [Empirical Design in Reinforcement Learning](https://mlsanity.com/paper/2304.01315/) (2023)
- [Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learning](https://mlsanity.com/paper/2303.17503/) (2023)
- [The Difficulty of Passive Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2110.14020/) (2021)
- [Atari-5: Distilling the Arcade Learning Environment down to Five Games](https://mlsanity.com/paper/2210.02019/) (2022)
- [MICo: Improved representations via sampling-based state similarity for Markov decision processes](https://mlsanity.com/paper/2106.08229/) (2021)
- [In value-based deep reinforcement learning, a pruned network is a good network](https://mlsanity.com/paper/2402.12479/) (2024)
- [Small batch deep reinforcement learning](https://mlsanity.com/paper/2310.03882/) (2023)
- [Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo](https://mlsanity.com/paper/2305.18246/) (2023)
- [On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.01648/) (2021)
- [Towards Robust Offline Reinforcement Learning under Diverse Data Corruption](https://mlsanity.com/paper/2310.12955/) (2023)
- [Adaptive Rational Activations to Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09407/) (2021)
- [Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem](https://mlsanity.com/paper/2402.02868/) (2024)
- [Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning](https://mlsanity.com/paper/2506.15544/) (2025)
- [Prioritizing Samples in Reinforcement Learning with Reducible Loss](https://mlsanity.com/paper/2208.10483/) (2022)
- [On the consistency of hyper-parameter selection in value-based deep reinforcement learning](https://mlsanity.com/paper/2406.17523/) (2024)
- [Mixture of Experts in a Mixture of RL settings](https://mlsanity.com/paper/2406.18420/) (2024)
- [Proper Reuse of Image Classification Features Improves Object Detection](https://mlsanity.com/paper/2204.00484/) (2022)
- [PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping](https://mlsanity.com/paper/2312.12065/) (2023)
- [Large Batch Experience Replay](https://mlsanity.com/paper/2110.01528/) (2021)
- [Robots Enact Malignant Stereotypes](https://mlsanity.com/paper/2207.11569/) (2022)
- [Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL](https://mlsanity.com/paper/2410.01930/) (2024)

## Abstract (arXiv)

Since the introduction of DQN, a vast majority of reinforcement learning research has focused on reinforcement learning with deep neural networks as function approximators. New methods are typically evaluated on a set of environments that have now become standard, such as Atari 2600 games. While these benchmarks help standardize evaluation, their computational cost has the unfortunate side effect of widening the gap between those with ample access to computational resources, and those without. In this work we argue that, despite the community's emphasis on large-scale environments, the traditional small-scale environments can still yield valuable scientific insights and can help reduce the barriers to entry for underprivileged communities. To substantiate our claims, we empirically revisit the paper which introduced the Rainbow algorithm [Hessel et al., 2018] and present some new insights into the algorithms used by Rainbow.