# Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2506.17204 · HTML: https://mlsanity.com/paper/2506.17204/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 52 of its 58 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 52)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Stop Regressing: Training Value Functions via Classification for Scalable Deep RL](https://mlsanity.com/paper/2403.03950/) (2024)
- [Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2010.14498/) (2020)
- [Understanding and Preventing Capacity Loss in Reinforcement Learning](https://mlsanity.com/paper/2204.09560/) (2022)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [When to use parametric models in reinforcement learning?](https://mlsanity.com/paper/1906.05243/) (2019)
- [Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control](https://mlsanity.com/paper/2405.16158/) (2024)
- [CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity](https://mlsanity.com/paper/1902.05605/) (2019)
- [Better Exploration with Optimistic Actor-Critic](https://mlsanity.com/paper/1910.12807/) (2019)
- [Disentangling the Causes of Plasticity Loss in Neural Networks](https://mlsanity.com/paper/2402.18762/) (2024)
- [The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training](https://mlsanity.com/paper/2202.02643/) (2022)

## Built on by (most cited first, 11 of 11)

- [XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning](https://mlsanity.com/paper/2509.25174/) (2025)
- [Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents](https://mlsanity.com/paper/2510.13704/) (2025)
- [Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning](https://mlsanity.com/paper/2510.01656/) (2025)
- [What Makes Value Learning Efficient in Residual Reinforcement Learning?](https://mlsanity.com/paper/2602.10539/) (2026)
- [Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2510.12096/) (2025)
- [Learning Quantized Continuous Controllers for Integer Hardware](https://mlsanity.com/paper/2511.07046/) (2025)
- [Topology-Aware Revival for Efficient Sparse Training](https://mlsanity.com/paper/2602.04166/) (2026)
- [RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratch](https://mlsanity.com/paper/2602.14578/) (2026)
- [Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria](https://mlsanity.com/paper/2604.10560/) (2026)
- [Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2604.18978/) (2026)
- [Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic](https://mlsanity.com/paper/2609.32819/) (2026)

## Abstract (arXiv)

Effectively scaling up deep reinforcement learning models has proven notoriously difficult due to network pathologies during training, motivating various targeted interventions such as periodic reset and architectural advances such as layer normalization. Instead of pursuing more complex modifications, we show that introducing static network sparsity alone can unlock further scaling potential beyond their dense counterparts with state-of-the-art architectures. This is achieved through simple one-shot random pruning, where a predetermined percentage of network weights are randomly removed once before training. Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity and stronger resistance to optimization challenges like plasticity loss and gradient interference. We further extend our evaluation to visual and streaming RL scenarios, demonstrating the consistent benefits of network sparsity.