# D2RL: Deep Dense Architectures in Reinforcement Learning

arXiv: https://arxiv.org/abs/2010.09163 · HTML: https://mlsanity.com/paper/2010.09163/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 30 of 42)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Revisiting the Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2305.09836/) (2023)
- [S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning](https://mlsanity.com/paper/2103.06326/) (2021)
- [Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes](https://mlsanity.com/paper/2211.15144/) (2022)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [In value-based deep reinforcement learning, a pruned network is a good network](https://mlsanity.com/paper/2402.12479/) (2024)
- [Training Larger Networks for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.07920/) (2021)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo](https://mlsanity.com/paper/2305.18246/) (2023)
- [Policy Architectures for Compositional Generalization in Control](https://mlsanity.com/paper/2203.05960/) (2022)
- [Reinforcement Learning with Latent Flow](https://mlsanity.com/paper/2101.01857/) (2021)
- [Towards Deeper Deep Reinforcement Learning with Spectral Normalization](https://mlsanity.com/paper/2106.01151/) (2021)
- [Overcoming the Spectral Bias of Neural Value Approximation](https://mlsanity.com/paper/2206.04672/) (2022)
- [Functional Regularization for Reinforcement Learning via Learned Fourier Features](https://mlsanity.com/paper/2112.03257/) (2021)
- [Fast and Data-Efficient Training of Rainbow: an Experimental Study on Atari](https://mlsanity.com/paper/2111.10247/) (2021)
- [Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulation](https://mlsanity.com/paper/2311.04060/) (2023)
- [A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control](https://mlsanity.com/paper/2507.02712/) (2025)
- [Transformer in Transformer as Backbone for Deep Reinforcement Learning](https://mlsanity.com/paper/2212.14538/) (2022)
- [RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigation](https://mlsanity.com/paper/2502.02054/) (2025)
- [Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2503.05546/) (2025)
- [PDiT: Interleaving Perception and Decision-making Transformers for Deep Reinforcement Learning](https://mlsanity.com/paper/2312.15863/) (2023)
- [Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym](https://mlsanity.com/paper/2310.00077/) (2023)
- [Task and Domain Adaptive Reinforcement Learning for Robot Control](https://mlsanity.com/paper/2404.18713/) (2024)
- [Reinforced Multi-teacher Knowledge Distillation for Efficient General Image Forgery Detection and Localization](https://mlsanity.com/paper/2504.05224/) (2025)
- [Honey, I Shrunk The Actor: A Case Study on Preserving Performance with Smaller Actors in Actor-Critic RL](https://mlsanity.com/paper/2102.11893/) (2021)
- [Reinforcement Learning with Formal Performance Metrics for Quadcopter Attitude Control under Non-nominal Contexts](https://mlsanity.com/paper/2107.12942/) (2021)
- [Leveraging Demonstrations with Latent Space Priors](https://mlsanity.com/paper/2210.14685/) (2022)

## Abstract (arXiv)

While improvements in deep learning architectures have played a crucial role in improving the state of supervised and unsupervised learning in computer vision and natural language processing, neural network architecture choices for reinforcement learning remain relatively under-explored. We take inspiration from successful architectural choices in computer vision and generative modelling, and investigate the use of deeper networks and dense connections for reinforcement learning on a variety of simulated robotic learning benchmark environments. Our findings reveal that current methods benefit significantly from dense connections and deeper networks, across a suite of manipulation and locomotion tasks, for both proprioceptive and image-based observations. We hope that our results can serve as a strong baseline and further motivate future research into neural network architectures for reinforcement learning. The project website with code is at this link https://sites.google.com/view/d2rl/home.