# Robust Pruning at Initialization

arXiv: https://arxiv.org/abs/2002.08797 · HTML: https://mlsanity.com/paper/2002.08797/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 30 of 31)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning](https://mlsanity.com/paper/2110.11395/) (2021)
- [NOLA: Compressing LoRA using Linear Combination of Random Basis](https://mlsanity.com/paper/2310.02556/) (2023)
- [On the infinite-depth limit of finite-width neural networks](https://mlsanity.com/paper/2210.00688/) (2022)
- [Width and Depth Limits Commute in Residual Networks](https://mlsanity.com/paper/2302.00453/) (2023)
- [Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning](https://mlsanity.com/paper/2209.14624/) (2022)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [A Unified Framework for Soft Threshold Pruning](https://mlsanity.com/paper/2302.13019/) (2023)
- [Regularization in ResNet with Stochastic Depth](https://mlsanity.com/paper/2106.03091/) (2021)
- [Field theory for optimal signal propagation in ResNets](https://mlsanity.com/paper/2305.07715/) (2023)
- [Ex uno plures: Splitting One Model into an Ensemble of Subnetworks](https://mlsanity.com/paper/2106.04767/) (2021)
- [On the Initialisation of Wide Low-Rank Feedforward Neural Networks](https://mlsanity.com/paper/2301.13710/) (2023)
- [Magnitude Attention-based Dynamic Pruning](https://mlsanity.com/paper/2306.05056/) (2023)
- [Deep Neural Network Initialization with Sparsity Inducing Activations](https://mlsanity.com/paper/2402.16184/) (2024)
- [Lottery Tickets with Nonzero Biases](https://mlsanity.com/paper/2110.11150/) (2021)
- [MCNC: Manifold-Constrained Reparameterization for Neural Compression](https://mlsanity.com/paper/2406.19301/) (2024)
- [On the Stability of the Jacobian Matrix in Deep Neural Networks](https://mlsanity.com/paper/2506.08764/) (2025)
- [Bayesian Optimization with Clustering and Rollback for CNN Auto Pruning](https://mlsanity.com/paper/2109.10591/) (2021)
- [Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning](https://mlsanity.com/paper/2110.11804/) (2021)
- [Zeroth-Order Topological Insights into Iterative Magnitude Pruning](https://mlsanity.com/paper/2206.06563/) (2022)
- [One-shot Network Pruning at Initialization with Discriminative Image Patches](https://mlsanity.com/paper/2209.05683/) (2022)
- [LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratch](https://mlsanity.com/paper/2309.14157/) (2023)
- [PRANC: Pseudo RAndom Networks for Compacting deep models](https://mlsanity.com/paper/2206.08464/) (2022)
- [Commutative Width and Depth Scaling in Deep Neural Networks](https://mlsanity.com/paper/2310.01683/) (2023)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows](https://mlsanity.com/paper/2605.18870/) (2026)
- [Convolutional Network Fabric Pruning With Label Noise](https://mlsanity.com/paper/2202.07268/) (2022)
- [Neural Network Panning: Screening the Optimal Sparse Network Before Training](https://mlsanity.com/paper/2209.13378/) (2022)
- [Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers](https://mlsanity.com/paper/2306.09863/) (2023)
- [Sparsest Models Elude Pruning: An Expos'e of Pruning's Current Capabilities](https://mlsanity.com/paper/2407.04075/) (2024)

## Abstract (arXiv)

Overparameterized Neural Networks (NN) display state-of-the-art performance. However, there is a growing need for smaller, energy-efficient, neural networks tobe able to use machine learning applications on devices with limited computational resources. A popular approach consists of using pruning techniques. While these techniques have traditionally focused on pruning pre-trained NN (LeCun et al.,1990; Hassibi et al., 1993), recent work by Lee et al. (2018) has shown promising results when pruning at initialization. However, for Deep NNs, such procedures remain unsatisfactory as the resulting pruned networks can be difficult to train and, for instance, they do not prevent one layer from being fully pruned. In this paper, we provide a comprehensive theoretical analysis of Magnitude and Gradient based pruning at initialization and training of sparse architectures. This allows us to propose novel principled approaches which we validate experimentally on a variety of NN architectures.