# Pruning via Iterative Ranking of Sensitivity Statistics

arXiv: https://arxiv.org/abs/2006.00896 · HTML: https://mlsanity.com/paper/2006.00896/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 26 of 26)

- [The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training](https://mlsanity.com/paper/2202.02643/) (2022)
- [Sparse Training via Boosting Pruning Plasticity with Neuroregeneration](https://mlsanity.com/paper/2106.10404/) (2021)
- [Most Activation Functions Can Win the Lottery Without Excessive Depth](https://mlsanity.com/paper/2205.02321/) (2022)
- [Winning the Lottery Ahead of Time: Efficient Early Network Pruning](https://mlsanity.com/paper/2206.10451/) (2022)
- [Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs](https://mlsanity.com/paper/2203.07808/) (2022)
- [Convolutional and Residual Networks Provably Contain Lottery Tickets](https://mlsanity.com/paper/2205.02343/) (2022)
- [Sparse maximal update parameterization: A holistic approach to sparse training dynamics](https://mlsanity.com/paper/2405.15743/) (2024)
- [Plant 'n' Seek: Can You Find the Winning Ticket?](https://mlsanity.com/paper/2111.11153/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [PAD-Net: An Efficient Framework for Dynamic Networks](https://mlsanity.com/paper/2211.05528/) (2022)
- [Structurally Prune Anything: Any Architecture, Any Framework, Any Time](https://mlsanity.com/paper/2403.18955/) (2024)
- [LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification](https://mlsanity.com/paper/2204.02965/) (2022)
- [COPS: Controlled Pruning Before Training Starts](https://mlsanity.com/paper/2107.12673/) (2021)
- [Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons](https://mlsanity.com/paper/2403.07688/) (2024)
- [FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Training](https://mlsanity.com/paper/2506.10035/) (2025)
- [Lottery Tickets with Nonzero Biases](https://mlsanity.com/paper/2110.11150/) (2021)
- [Pruning for Feature-Preserving Circuits in CNNs](https://mlsanity.com/paper/2206.01627/) (2022)
- [Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection](https://mlsanity.com/paper/2310.13183/) (2023)
- [RGP: Neural Network Pruning through Its Regular Graph Structure](https://mlsanity.com/paper/2110.15192/) (2021)
- [On the Robustness and Anomaly Detection of Sparse Neural Networks](https://mlsanity.com/paper/2207.04227/) (2022)
- [Graph is all you need? Lightweight data-agnostic neural architecture search without training](https://mlsanity.com/paper/2405.01306/) (2024)
- [Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning](https://mlsanity.com/paper/2411.13545/) (2024)
- [Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis](https://mlsanity.com/paper/2106.11542/) (2021)
- [A Generalization of Continuous Relaxation in Structured Pruning](https://mlsanity.com/paper/2308.14605/) (2023)
- [Learning effective pruning at initialization from iterative pruning](https://mlsanity.com/paper/2408.14757/) (2024)
- [BaCP: Backbone Contrastive Pruning for Preserving Representations in Extremely Sparse Neural Networks](https://mlsanity.com/paper/2610.02524/) (2026)

## Abstract (arXiv)

With the introduction of SNIP [arXiv:1810.02340v2], it has been demonstrated that modern neural networks can effectively be pruned before training. Yet, its sensitivity criterion has since been criticized for not propagating training signal properly or even disconnecting layers. As a remedy, GraSP [arXiv:2002.07376v1] was introduced, compromising on simplicity. However, in this work we show that by applying the sensitivity criterion iteratively in smaller steps - still before training - we can improve its performance without difficult implementation. As such, we introduce 'SNIP-it'. We then demonstrate how it can be applied for both structured and unstructured pruning, before and/or during training, therewith achieving state-of-the-art sparsity-performance trade-offs. That is, while already providing the computational benefits of pruning in the training process from the start. Furthermore, we evaluate our methods on robustness to overfitting, disconnection and adversarial attacks as well.