mlsanityData to 14 Oct 2026

Dynamic Sparse Training for Deep Reinforcement Learning

arXiv 2106.04217 ↗First posted 8 Jun 2021cs.LG cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training time for dense neural networks to achieve good performance. Hence, prohibitive computation and memory resources are consumed. Recently, learning efficient DRL agents has received increasing attention. Yet, current methods focus on accelerating inference time. In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process. The proposed approach trains a sparse neural network from scratch and dynamically adapts its topology to the changing data distribution during training. Experiments on continuous control tasks show that our dynamic sparse agents achieve higher performance than the equivalent dense methods, reduce the parameter count and floating-point operations (FLOPs) by 50%, and have a faster learning speed that enables reaching the performance of dense agents with 40-50% reduction in the training steps.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 54 of its 83 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

54 of 83 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  6. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  7. 2019Importance Estimation for Neural Network Pruningcited by 541
  8. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  9. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  10. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 461
  11. 2017Deep Reinforcement Learning: An Overviewcited by 333
  12. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  13. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  14. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  15. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 233
  16. 2013Predicting Parameters in Deep Learningcited by 224
  17. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 221
  18. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 221
  19. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  20. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
  21. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  22. 2020The Hardware Lotterycited by 160
  23. 2017Deep Rewiring: Training very sparse deep networkscited by 139
  24. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  25. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  26. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  27. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  28. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  29. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  30. 2020SpaceNet: Make Free Space For Continual Learningcited by 53
  31. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
  32. 2020Sparse Weight Activation Trainingcited by 45
  33. 2018Multi-objective Evolutionary Federated Learningcited by 42
  34. 2020PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learningcited by 30
  35. 2020Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoderscited by 28
  36. 2021Selfish Sparse RNN Trainingcited by 26
  37. 2021On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learningcited by 24
  38. 2020Topological Insights into Sparse Neural Networkscited by 22

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 12; 2024: 16; 2025: 11; 2026: 4.

49 papers.

Sort:

By area:

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  3. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  4. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  5. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  6. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  7. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  8. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  9. 2024Streaming Deep Reinforcement Learning Finally Workscited by 25
  10. 2022Dynamic Sparse Network for Time Series Classification: Learning What to "see''cited by 22
  11. 2021Single-Shot Pruning for Offline Reinforcement Learningcited by 21
  12. 2024Mixture of Experts in a Mixture of RL settingscited by 21
37 more
  1. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 18
  2. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 17
  3. 2022Where to Pay Attention in Sparse Training for Feature Selection?cited by 15
  4. 2023Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learningcited by 14
  5. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  6. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13
  7. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  8. 2021Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networkscited by 8
  9. 2023Supervised Feature Selection with Neuron Evolution in Sparse Neural Networkscited by 7
  10. 2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learningcited by 6
  11. 2025Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learningcited by 4
  12. 2023Towards model-free RL algorithms that scale well with unstructured datacited by 3
  13. 2024Unveiling the Power of Sparse Neural Networks for Feature Selectioncited by 3
  14. 2025On the Interplay Between Sparsity and Training in Deep Reinforcement Learningcited by 3
  15. 2024Gradient-Congruity Guided Federated Sparse Trainingcited by 2
  16. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Modelscited by 2
  17. 2023SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Trainingcited by 1
  18. 2023Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformerscited by 1
  19. 2024Evasive Active Hypothesis Testing with Deep Neuroevolution: The Single- and Multi-Agent Casescited by 1
  20. 2024Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real Worldcited by 1
  21. 2024SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learningcited by 1
  22. 2024Hadamard Representation: Scaffolding Performance Across Model-free RLcited by 1
  23. 2024Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Trainingcited by 1
  24. 2025Embedded Federated Feature Selection with Dynamic Sparse Training: Balancing Accuracy-Cost Tradeoffscited by 1
  25. 2025Sparse-to-Sparse Training of Diffusion Modelscited by 1
  26. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  27. 2025Frustratingly Easy Task-aware Pruning for Large Language Modelscited by 1
  28. 2023One is More: Diverse Perspectives within a Single Network for Efficient DRLcited by 0
  29. 2024Hard-Thresholding Meets Evolution Strategies in Reinforcement Learningcited by 0
  30. 2024Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsitycited by 0
  31. 2024Learning Emergence of Interaction Patterns across Independent RL Agents in Multi-Agent Environmentscited by 0
  32. 2025NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensemblingcited by 0
  33. 2025Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functionscited by 0
  34. 2026RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratchcited by 0
  35. 2026Sparse Masked Attention Policies for Reliable Generalizationcited by 0
  36. 2026WINFlowNets: Warm-up Integrated Networks Training of Generative Flow Networks for Robotics and Machine Fault Adaptationcited by 0
  37. 2026Higher Resolution, Better Generalization: Unlocking Visual Scaling in Deep Reinforcement Learningcited by 0