mlsanityData to 14 Oct 2026

Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

arXiv 2506.17204 ↗First posted 20 Jun 2025cs.LG cs.AIcited by 11

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Effectively scaling up deep reinforcement learning models has proven notoriously difficult due to network pathologies during training, motivating various targeted interventions such as periodic reset and architectural advances such as layer normalization. Instead of pursuing more complex modifications, we show that introducing static network sparsity alone can unlock further scaling potential beyond their dense counterparts with state-of-the-art architectures. This is achieved through simple one-shot random pruning, where a predetermined percentage of network weights are randomly removed once before training. Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity and stronger resistance to optimization challenges like plasticity loss and gradient interference. We further extend our evaluation to visual and streaming RL scenarios, demonstrating the consistent benefits of network sparsity.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 52 of its 58 references that are in the map. First posted on arXiv 20 Jun 2025.

Builds on

52 of 58 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Plasticity Loss in Deep Reinforcement Learning: A Survey7 Nov
  2. 2024Streaming Deep Reinforcement Learning Finally Works18 Oct
  3. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  4. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?9 Jul
  5. 2024Normalization and effective learning rates in reinforcement learning1 Jul
  6. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning29 May
  7. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  8. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar
  9. 2024Disentangling the Causes of Plasticity Loss in Neural Networks29 Feb
  10. 2024In value-based deep reinforcement learning, a pruned network is a good network19 Feb
  11. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 41

  1. 2016Layer Normalizationcited by 5,669
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2018DeepMind Control Suitecited by 956
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  10. 2019Model-Based Reinforcement Learning for Ataricited by 502
  11. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
29 more
  1. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  2. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  3. 2023From Sparse to Soft Mixtures of Expertscited by 248
  4. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  5. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  6. 2023Understanding plasticity in neural networkscited by 174
  7. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  8. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  9. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  10. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  11. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  12. 2019When to use parametric models in reinforcement learning?cited by 118
  13. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  14. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  15. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  16. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  17. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  18. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  19. 2020Scalable Transfer Learning with Expert Modelscited by 49
  20. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  21. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  22. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  23. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  24. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  25. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 28
  26. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  27. 2021On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learningcited by 24
  28. 2021Single-Shot Pruning for Offline Reinforcement Learningcited by 21
  29. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  2. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  3. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
  4. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 4
  5. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  6. 2025Learning Quantized Continuous Controllers for Integer Hardwarecited by 1
  7. 2026Topology-Aware Revival for Efficient Sparse Trainingcited by 0
  8. 2026RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratchcited by 0
  9. 2026Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibriacited by 0
  10. 2026Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learningcited by 0
  11. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Online Training and Pruning of Deep Reinforcement Learning Networks
  2. 2026Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning
  3. 2025On the Interplay Between Sparsity and Training in Deep Reinforcement Learning
  4. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
  5. 2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning
  6. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
  7. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models
  8. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning