mlsanityData to 7 Oct 2026

Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers

arXiv 2005.06870 ↗First posted 14 May 2020cs.LG stat.MLcited by 66

Not on the map. Why?

Abstract (arXiv)

We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly find the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds. These thresholds can have fine-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same number of training epochs as dense models. Dynamic Sparse Training achieves the state of the art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence for the effectiveness and efficiency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 9; 2023: 16; 2024: 15; 2025: 8; 2026: 6.

66 papers.

Sort:

By area:

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  3. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  4. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  5. 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 38
  6. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  7. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  8. 2021Selfish Sparse RNN Trainingcited by 26
  9. 2021Single-Shot Pruning for Offline Reinforcement Learningcited by 21
  10. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  11. 2024Graph Sparsification via Mixture of Graphscited by 17
  12. 2022Compression-aware Training of Neural Networks using Frank-Wolfecited by 16
38 more
  1. 2021Truly Sparse Neural Networks at Scalecited by 15
  2. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  3. 2022Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruningcited by 13
  4. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13
  5. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 13
  6. 2021Sparse Training Theory for Scalable and Efficient Agentscited by 12
  7. 2021How I Learned to Stop Worrying and Love Retrainingcited by 12
  8. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  9. 2024ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networkscited by 11
  10. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  11. 2024Two Heads Are Better Than One: Boosting Graph Sparse Training via Semantic and Topological Awarenesscited by 10
  12. 2024SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overheadcited by 9
  13. 2023Balanced Training for Sparse GANscited by 8
  14. 2024DynST: Dynamic Sparse Training for Resource-Constrained Spatio-Temporal Forecastingcited by 7
  15. 2024SymbolNet: Neural Symbolic Regression with Adaptive Dynamic Pruning for Compressioncited by 6
  16. 2024COPAL: Continual Pruning in Large Language Generative Modelscited by 6
  17. 2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learningcited by 6
  18. 2021Dynamic Probabilistic Pruning: A general framework for hardware-constrained pruning at different granularitiescited by 5
  19. 2023Magnitude Attention-based Dynamic Pruningcited by 5
  20. 2024SCATTER: Algorithm-Circuit Co-Sparse Photonic Accelerator with Thermal-Tolerant, Power-Efficient In-situ Light Redistributioncited by 5
  21. 2022AdaptCL: Adaptive Continual Learning for Tackling Heterogeneity in Sequential Datasetscited by 4
  22. 2023Neural Network Pruning by Gradient Descentcited by 4
  23. 2021Search Spaces for Neural Model Trainingcited by 3
  24. 2023Boosting Convolutional Neural Networks with Middle Spectrum Grouped Convolutioncited by 3
  25. 2023Resource Efficient Neural Networks Using Hessian Based Pruningcited by 3
  26. 2023Compressing the Backward Pass of Large-Scale Neural Architectures by Structured Activation Pruningcited by 3
  27. 2024Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Modelscited by 3
  28. 2025On the Interplay Between Sparsity and Training in Deep Reinforcement Learningcited by 3
  29. 2021A Deeper Look into Convolutions via Eigenvalue-based Pruningcited by 2
  30. 2023AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networkscited by 2
  31. 2023Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruningcited by 2
  32. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  33. 2024Learning in Order! A Sequential Strategy to Learn Invariant Features for Multimodal Sentiment Analysiscited by 2
  34. 2021Exploiting Elasticity in Tensor Ranks for Compressing Neural Networkscited by 1
  35. 2021RGP: Neural Network Pruning through Its Regular Graph Structurecited by 1
  36. 2022MIME: Adapting a Single Neural Network for Multi-task Inference with Memory-efficient Dynamic Pruningcited by 1
  37. 2022MiNL: Micro-images based Neural Representation for Light Fieldscited by 1
  38. 2024Pruning neural network models for gene regulatory dynamics using data and domain knowledgecited by 1

50 most cited of 66 All 66 (JSON)