mlsanityData to 7 Oct 2026

Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win

arXiv 2010.03533 ↗First posted 7 Oct 2020cs.LG cs.CVcited by 77

Not on the map. Why?

Abstract (arXiv)

Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction of the compute/storage for inference, and also have the potential to enable efficient training. However, naively training unstructured sparse NNs from random initialization results in significantly worse generalization, with the notable exceptions of Lottery Tickets (LTs) and Dynamic Sparse Training (DST). Through our analysis of gradient flow during training we attempt to answer: (1) why training unstructured sparse networks from random initialization performs poorly and; (2) what makes LTs and DST the exceptions? We show that sparse NNs have poor gradient flow at initialization and demonstrate the importance of using sparsity-aware initialization. Furthermore, we find that DST methods significantly improve gradient flow during training over traditional sparse training methods. Finally, we show that LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from - however, this comes at the cost of learning novel solutions.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 12; 2023: 17; 2024: 15; 2025: 14; 2026: 3.

77 papers.

Sort:

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  3. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  4. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  5. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  6. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  7. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  8. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  9. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  10. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  11. 2023Dynamic Sparse Training with Structured Sparsitycited by 36
  12. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
38 more
  1. 2022Training Your Sparse Neural Network Better with Any Maskcited by 26
  2. 2022Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraintscited by 25
  3. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  4. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  5. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  6. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  7. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  8. 2022Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networkscited by 18
  9. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 16
  10. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  11. 2021Truly Sparse Neural Networks at Scalecited by 15
  12. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  13. 2021Sparse Training Theory for Scalable and Efficient Agentscited by 12
  14. 2021Towards Structured Dynamic Sparse Pre-Training of BERTcited by 12
  15. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  16. 2021Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Wincited by 11
  17. 2022Lottery Pools: Winning More by Interpolating Tickets without Increasing Training or Inference Costcited by 11
  18. 2024Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustnesscited by 11
  19. 2022Training Recipe for N:M Structured Sparsity with Decaying Pruning Maskcited by 10
  20. 2024Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generationcited by 9
  21. 2021Universality of Winning Tickets: A Renormalization Group Perspectivecited by 8
  22. 2023Supervised Feature Selection with Neuron Evolution in Sparse Neural Networkscited by 7
  23. 2024Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendationcited by 7
  24. 2025Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuningcited by 7
  25. 2021Spending Your Winning Lottery Better After Drawing Itcited by 5
  26. 2023Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correctioncited by 5
  27. 2024Improving Local Training in Federated Learning via Temperature Scalingcited by 5
  28. 2021How much pre-training is enough to discover a good subnetwork?cited by 4
  29. 2023Neural Network Pruning by Gradient Descentcited by 4
  30. 2021Search Spaces for Neural Model Trainingcited by 3
  31. 2022Strong Lottery Ticket Hypothesis with varepsilon--perturbationcited by 3
  32. 2023Benchmarking Adversarial Robustness of Compressed Deep Learning Modelscited by 3
  33. 2024Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initializationcited by 3
  34. 2024Navigating Extremes: Dynamic Sparsity in Large Output Spacescited by 3
  35. 2025On the Stability of the Jacobian Matrix in Deep Neural Networkscited by 3
  36. 2023Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruningcited by 2
  37. 2023Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representationscited by 2
  38. 2025Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetrycited by 2

50 most cited of 77 All 77 (JSON)