mlsanityData to 7 Oct 2026

Robust Pruning at Initialization

arXiv 2002.08797 ↗First posted 19 Feb 2020stat.ML cs.CV cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Overparameterized Neural Networks (NN) display state-of-the-art performance. However, there is a growing need for smaller, energy-efficient, neural networks tobe able to use machine learning applications on devices with limited computational resources. A popular approach consists of using pruning techniques. While these techniques have traditionally focused on pruning pre-trained NN (LeCun et al.,1990; Hassibi et al., 1993), recent work by Lee et al. (2018) has shown promising results when pruning at initialization. However, for Deep NNs, such procedures remain unsatisfactory as the resulting pruned networks can be difficult to train and, for instance, they do not prevent one layer from being fully pruned. In this paper, we provide a comprehensive theoretical analysis of Magnitude and Gradient based pruning at initialization and training of sparse architectures. This allows us to propose novel principled approaches which we validate experimentally on a variety of NN architectures.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 10; 2024: 3; 2025: 3; 2026: 1.

31 papers.

Sort:

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2021SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruningcited by 30
  3. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 28
  4. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  5. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  6. 2022Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruningcited by 13
  7. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  8. 2023A Unified Framework for Soft Threshold Pruningcited by 12
  9. 2021Regularization in ResNet with Stochastic Depthcited by 7
  10. 2023Field theory for optimal signal propagation in ResNetscited by 6
  11. 2021Ex uno plures: Splitting One Model into an Ensemble of Subnetworkscited by 5
  12. 2023On the Initialisation of Wide Low-Rank Feedforward Neural Networkscited by 5
19 more
  1. 2023Magnitude Attention-based Dynamic Pruningcited by 5
  2. 2024Deep Neural Network Initialization with Sparsity Inducing Activationscited by 5
  3. 2021Lottery Tickets with Nonzero Biasescited by 4
  4. 2024MCNC: Manifold-Constrained Reparameterization for Neural Compressioncited by 4
  5. 2025On the Stability of the Jacobian Matrix in Deep Neural Networkscited by 3
  6. 2021Bayesian Optimization with Clustering and Rollback for CNN Auto Pruningcited by 2
  7. 2021Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learningcited by 2
  8. 2022Zeroth-Order Topological Insights into Iterative Magnitude Pruningcited by 2
  9. 2022One-shot Network Pruning at Initialization with Discriminative Image Patchescited by 2
  10. 2023LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratchcited by 2
  11. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  12. 2025Onboard Optimization and Learning: A Surveycited by 1
  13. 2022Convolutional Network Fabric Pruning With Label Noisecited by 0
  14. 2022PRANC: Pseudo RAndom Networks for Compacting deep modelscited by 0
  15. 2022Neural Network Panning: Screening the Optimal Sparse Network Before Trainingcited by 0
  16. 2023Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solverscited by 0
  17. 2024Sparsest Models Elude Pruning: An Expos'e of Pruning's Current Capabilitiescited by 0
  18. 2025Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learningcited by 0
  19. 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flowscited by 0