# Dynamic Sparse Training for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2106.04217 · HTML: https://mlsanity.com/paper/2106.04217/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 83 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 54)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker](https://mlsanity.com/paper/1701.01724/) (2017)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures](https://mlsanity.com/paper/1607.03250/) (2016)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)

## Built on by (most cited first, 30 of 49)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Mixtures of Experts Unlock Parameter Scaling for Deep RL](https://mlsanity.com/paper/2402.08609/) (2024)
- [The State of Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2206.10369/) (2022)
- [Maintaining Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2306.13812/) (2023)
- [In value-based deep reinforcement learning, a pruned network is a good network](https://mlsanity.com/paper/2402.12479/) (2024)
- [RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch](https://mlsanity.com/paper/2205.15043/) (2022)
- [Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers](https://mlsanity.com/paper/2302.02596/) (2023)
- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [Dynamic Sparse Network for Time Series Classification: Learning What to "see''](https://mlsanity.com/paper/2212.09840/) (2022)
- [Single-Shot Pruning for Offline Reinforcement Learning](https://mlsanity.com/paper/2112.15579/) (2021)
- [Mixture of Experts in a Mixture of RL settings](https://mlsanity.com/paper/2406.18420/) (2024)
- [Neuroplastic Expansion in Deep Reinforcement Learning](https://mlsanity.com/paper/2410.07994/) (2024)
- [Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training](https://mlsanity.com/paper/2306.12230/) (2023)
- [Where to Pay Attention in Sparse Training for Feature Selection?](https://mlsanity.com/paper/2211.14627/) (2022)
- [Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2303.00912/) (2023)
- [Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL](https://mlsanity.com/paper/2410.01930/) (2024)
- [Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.06548/) (2023)
- [Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17204/) (2025)
- [Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks](https://mlsanity.com/paper/2110.05329/) (2021)
- [Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks](https://mlsanity.com/paper/2303.07200/) (2023)
- [Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning](https://mlsanity.com/paper/2503.01437/) (2025)
- [Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning](https://mlsanity.com/paper/2505.17749/) (2025)
- [Towards model-free RL algorithms that scale well with unstructured data](https://mlsanity.com/paper/2311.02215/) (2023)
- [Unveiling the Power of Sparse Neural Networks for Feature Selection](https://mlsanity.com/paper/2408.04583/) (2024)
- [On the Interplay Between Sparsity and Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2501.16729/) (2025)
- [Gradient-Congruity Guided Federated Sparse Training](https://mlsanity.com/paper/2405.01189/) (2024)
- [Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models](https://mlsanity.com/paper/2507.17107/) (2025)
- [SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Training](https://mlsanity.com/paper/2304.07488/) (2023)
- [Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers](https://mlsanity.com/paper/2305.18382/) (2023)

## Abstract (arXiv)

Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training time for dense neural networks to achieve good performance. Hence, prohibitive computation and memory resources are consumed. Recently, learning efficient DRL agents has received increasing attention. Yet, current methods focus on accelerating inference time. In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process. The proposed approach trains a sparse neural network from scratch and dynamically adapts its topology to the changing data distribution during training. Experiments on continuous control tasks show that our dynamic sparse agents achieve higher performance than the equivalent dense methods, reduce the parameter count and floating-point operations (FLOPs) by 50%, and have a faster learning speed that enables reaching the performance of dense agents with 40-50% reduction in the training steps.