mlsanityData to 7 Oct 2026

Recent Advances on Neural Network Pruning at Initialization

arXiv 2103.06460 ↗First posted 11 Mar 2021cs.LG cs.AI cs.CVcited by 41

Not on the map. Why?

Abstract (arXiv)

Neural network pruning typically removes connections or neurons from a pretrained converged model; while a new pruning paradigm, pruning at initialization (PaI), attempts to prune a randomly initialized network. This paper offers the first survey concentrated on this emerging pruning fashion. We first introduce a generic formulation of neural network pruning, followed by the major classic pruning topics. Then, as the main body of this paper, a thorough and structured literature review of PaI methods is presented, consisting of two major tracks (sparse training and sparse selection). Finally, we summarize the surge of PaI compared to PaT and discuss the open problems. Apart from the dedicated literature review, this paper also offers a code base for easy sanity-checking and benchmarking of different PaI methods.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 48 of its 79 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

48 of 79 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  8. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  10. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  11. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  12. 2018Rethinking the Value of Network Pruningcited by 595
36 more
  1. 2020What is the State of Neural Network Pruning?cited by 578
  2. 2019Importance Estimation for Neural Network Pruningcited by 531
  3. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  4. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  5. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  6. 2019The State of Sparsity in Deep Neural Networkscited by 471
  7. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  8. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
  9. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  10. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  11. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  12. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  13. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  14. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  15. 2020Supermasks in Superpositioncited by 223
  16. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  17. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  18. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  19. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  20. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  21. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  22. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  23. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  24. 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
  25. 2020The Early Phase of Neural Network Trainingcited by 114
  26. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  27. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  28. 2020Neural Pruning via Growing Regularizationcited by 97
  29. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  30. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  31. 2020Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficientcited by 81
  32. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  33. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  34. 2020Logarithmic Pruning is All You Needcited by 71
  35. 2020Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpotcited by 66
  36. 2018Structured Pruning for Efficient ConvNets via Incremental Regularizationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 11; 2024: 9; 2025: 15; 2026: 1.

41 papers.

Sort:

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2025SepPrune: Structured Pruning for Efficient Deep Speech Separationcited by 31
  3. 2022Trainability Preserving Neural Pruningcited by 16
  4. 2023Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruningcited by 16
  5. 2023Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networkscited by 15
  6. 2023Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learningcited by 13
  7. 2023Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theorycited by 12
  8. 2022Parameter-Efficient Masking Networkscited by 9
  9. 2023PELA: Learning Parameter-Efficient Models with Low-Rank Approximationcited by 9
  10. 2023MixtureGrowth: Growing Neural Networks by Recombining Learned Parameterscited by 7
  11. 2024Is Oracle Pruning the True Oracle?cited by 7
  12. 2023Adaptive parameter sharing for multi-agent reinforcement learningcited by 6
29 more
  1. 2024EPSD: Early Pruning with Self-Distillation for Efficient Model Compressioncited by 6
  2. 2024Effective Layer Pruning Through Similarity Metric Perspectivecited by 6
  3. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6
  4. 2025Efficient LLMs with AMP: Attention Heads and MLP Pruningcited by 5
  5. 2023UPSCALE: Unconstrained Channel Pruningcited by 4
  6. 2024Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initializationcited by 3
  7. 2025Boosting Large Language Models with Mask Fine-Tuningcited by 3
  8. 2025On the Stability of the Jacobian Matrix in Deep Neural Networkscited by 3
  9. 2025One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compressioncited by 3
  10. 2022One-shot Network Pruning at Initialization with Discriminative Image Patchescited by 2
  11. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  12. 2025The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysiscited by 2
  13. 2023Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environmentscited by 1
  14. 2024Computer Vision Model Compression Techniques for Embedded Systems: A Surveycited by 1
  15. 2024A Comparative Study of Pruning Methods in Transformer-based Time Series Forecastingcited by 1
  16. 2025What Scalable Second-Order Information Knows for Pruning at Initializationcited by 1
  17. 2025Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrumcited by 1
  18. 2025Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruningcited by 1
  19. 2021Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysiscited by 0
  20. 2022Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networkscited by 0
  21. 2023Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Taskscited by 0
  22. 2024Layer Pruning with Consensus: A Triple-Win Solutioncited by 0
  23. 2025Optimize Incompatible Parameters through Compatibility-aware Knowledge Integrationcited by 0
  24. 2025Towards Symmetric Low-Rank Adapterscited by 0
  25. 2025Towards Scalable Lottery Ticket Networks using Genetic Algorithmscited by 0
  26. 2025Application of discrete Ricci curvature in pruning randomly wired neural networks: A case study with chest x-ray classification of COVID-19cited by 0
  27. 2025A Recovery Guarantee for Sparse Neural Networkscited by 0
  28. 2025Neural expressiveness for beyond importance model compressioncited by 0
  29. 2026Topology-Aware Revival for Efficient Sparse Trainingcited by 0