mlsanityData to 14 Oct 2026

Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training

arXiv 2306.12230 ↗First posted 21 Jun 2023cs.LG cs.AI cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network's sparse connectivity. While the growing criterion's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue, we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude-based pruning. The code is provided at https://github.com/alooow/fantastic_weights_paper

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 53 of its 66 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

53 of 66 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  12. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
38 more
  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  2. 2020What is the State of Neural Network Pruning?cited by 583
  3. 2019Importance Estimation for Neural Network Pruningcited by 541
  4. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  5. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  6. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  7. 2019The State of Sparsity in Deep Neural Networkscited by 476
  8. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  9. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  10. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  11. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  12. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  13. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 207
  14. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 182
  15. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  16. 2020The Hardware Lotterycited by 160
  17. 2017Deep Rewiring: Training very sparse deep networkscited by 139
  18. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  19. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  20. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  21. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 79
  22. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  23. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  24. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  25. 2017NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigmcited by 60
  26. 2019Discovering Neural Wiringscited by 56
  27. 2020SpaceNet: Make Free Space For Continual Learningcited by 53
  28. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  29. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
  30. 2022Rare Gems: Finding Lottery Tickets at Initializationcited by 34
  31. 2016A topological insight into restricted Boltzmann machinescited by 32
  32. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  33. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  34. 2020Topological Insights into Sparse Neural Networkscited by 22
  35. 2021Truly Sparse Neural Networks at Scalecited by 16
  36. 2021Sparse Training Theory for Scalable and Efficient Agentscited by 13
  37. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13
  38. 2019A Brain-inspired Algorithm for Training Highly Sparse Neural Networkscited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 6; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustnesscited by 11
  2. 2024SCAN: Bootstrapping Contrastive Pre-training for Data Efficiencycited by 6
  3. 2023Accelerating Learnt Video Codecs with Gradient Decay and Layer-wise Distillationcited by 4
  4. 2024Navigating Extremes: Dynamic Sparsity in Large Output Spacescited by 3
  5. 2025Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolutioncited by 2
  6. 2024A Comparative Study of Pruning Methods in Transformer-based Time Series Forecastingcited by 1
  7. 2025Sparse-to-Sparse Training of Diffusion Modelscited by 1
  8. 2024Mixed Sparsity Training: Achieving 4× FLOP Reduction for Transformer Pretrainingcited by 0
  9. 2024EntryPrune: Neural Network Feature Selection using First Impressionscited by 0
  10. 2025NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensemblingcited by 0
  11. 2025Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsitycited by 0
  12. 2025Neuro-inspired Ensemble-to-Ensemble Communication Primitives for Sparse and Efficient ANNscited by 0
5 more
  1. 2025A Recovery Guarantee for Sparse Neural Networkscited by 0
  2. 2026Exploring the impact of adaptive rewiring in Graph Neural Networkscited by 0
  3. 2026Rethinking Reservoir Pruning: A Dynamical Perspective for Echo State Networkscited by 0
  4. 2026HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classificationcited by 0
  5. 2026When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Trainingcited by 0