mlsanityData to 7 Oct 2026

Pruning Neural Networks at Initialization: Why are We Missing the Mark?

arXiv 2009.08576 ↗First posted 18 Sep 2020cs.LG cs.CV cs.NEcited by 181

Not on the map. Why?

Abstract (arXiv)

Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 41; 2023: 35; 2024: 42; 2025: 15; 2026: 12.

181 papers.

Sort:

By area:

  1. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  2. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2023EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingScited by 191
  5. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  6. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  7. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  8. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  9. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  10. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  11. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  12. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
38 more
  1. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  2. 2021Meta-Learning Sparse Implicit Neural Representationscited by 48
  3. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  4. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  5. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  6. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  7. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  8. 2022Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradientscited by 34
  9. 2022Rare Gems: Finding Lottery Tickets at Initializationcited by 34
  10. 2021Generalization bounds via distillationcited by 33
  11. 2023Towards provably efficient quantum algorithms for large-scale machine-learning modelscited by 32
  12. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  13. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  14. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  15. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  16. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  17. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  18. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  19. 2022Training Your Sparse Neural Network Better with Any Maskcited by 26
  20. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  21. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  22. 2024Spectral Graph Pruning Against Over-Squashing and Over-Smoothingcited by 23
  23. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  24. 2021Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?cited by 22
  25. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 22
  26. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  27. 2022Most Activation Functions Can Win the Lottery Without Excessive Depthcited by 21
  28. 2022Winning the Lottery Ahead of Time: Efficient Early Network Pruningcited by 21
  29. 2022Symmetric Pruning in Quantum Neural Networkscited by 20
  30. 2021Scaling Up Exact Neural Network Compression by ReLU Stabilitycited by 19
  31. 2021Neural Network Repair with Reachability Analysiscited by 19
  32. 2021An Operator Theoretic View on Pruning Deep Neural Networkscited by 18
  33. 2023Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AIcited by 17
  34. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 17
  35. 2021You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownershipcited by 16
  36. 2022Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNscited by 16
  37. 2022Convolutional and Residual Networks Provably Contain Lottery Ticketscited by 16
  38. 2023Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruningcited by 16

50 most cited of 181 All 181 (JSON)