# Progressive Skeletonization: Trimming more fat from a network at initialization

arXiv: https://arxiv.org/abs/2006.09081 · HTML: https://mlsanity.com/paper/2006.09081/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 30 of 64)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training](https://mlsanity.com/paper/2102.02887/) (2021)
- [The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training](https://mlsanity.com/paper/2202.02643/) (2022)
- [Sparse Training via Boosting Pruning Plasticity with Neuroregeneration](https://mlsanity.com/paper/2106.10404/) (2021)
- [The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter](https://mlsanity.com/paper/2306.03805/) (2023)
- [Structural Pruning via Latency-Saliency Knapsack](https://mlsanity.com/paper/2210.06659/) (2022)
- [When to Prune? A Policy towards Early Structural Pruning](https://mlsanity.com/paper/2110.12007/) (2021)
- [Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients](https://mlsanity.com/paper/2202.08132/) (2022)
- [NTK-SAP: Improving neural network pruning by aligning training dynamics](https://mlsanity.com/paper/2304.02840/) (2023)
- [Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!](https://mlsanity.com/paper/2303.02141/) (2023)
- [Selfish Sparse RNN Training](https://mlsanity.com/paper/2101.09048/) (2021)
- [Why Random Pruning Is All We Need to Start Sparse](https://mlsanity.com/paper/2210.02412/) (2022)
- [LEMON: Lossless model expansion](https://mlsanity.com/paper/2310.07999/) (2023)
- [Winning the Lottery Ahead of Time: Efficient Early Network Pruning](https://mlsanity.com/paper/2206.10451/) (2022)
- [OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition](https://mlsanity.com/paper/2409.13652/) (2024)
- [Adaptive Sharpness-Aware Pruning for Robust Sparse Networks](https://mlsanity.com/paper/2306.14306/) (2023)
- [Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs](https://mlsanity.com/paper/2203.07808/) (2022)
- [Efficient Stein Variational Inference for Reliable Distribution-lossless Network Pruning](https://mlsanity.com/paper/2212.03537/) (2022)
- [Sparse maximal update parameterization: A holistic approach to sparse training dynamics](https://mlsanity.com/paper/2405.15743/) (2024)
- [Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-off](https://mlsanity.com/paper/2211.16667/) (2022)
- [Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks](https://mlsanity.com/paper/2308.10438/) (2023)
- [Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity](https://mlsanity.com/paper/2107.02306/) (2021)
- [Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning](https://mlsanity.com/paper/2209.14624/) (2022)
- [MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning](https://mlsanity.com/paper/2404.05621/) (2024)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Soft Masking for Cost-Constrained Channel Pruning](https://mlsanity.com/paper/2211.02206/) (2022)
- [Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruning](https://mlsanity.com/paper/2406.01820/) (2024)
- [Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset](https://mlsanity.com/paper/2102.07655/) (2021)
- [PAD-Net: An Efficient Framework for Dynamic Networks](https://mlsanity.com/paper/2211.05528/) (2022)
- [HALP: Hardware-Aware Latency Pruning](https://mlsanity.com/paper/2110.10811/) (2021)

## Abstract (arXiv)

Recent studies have shown that skeletonization (pruning parameters) of networks textitat initialization provides all the practical benefits of sparsity both at inference and training time, while only marginally degrading their performance. However, we observe that beyond a certain level of sparsity (approx 95%), these approaches fail to preserve the network performance, and to our surprise, in many cases perform even worse than trivial random pruning. To this end, we propose an objective to find a skeletonized network with maximum em foresight connection sensitivity (FORCE) whereby the trainability, in terms of connection sensitivity, of a pruned network is taken into consideration. We then propose two approximate procedures to maximize our objective (1) Iterative SNIP: allows parameters that were unimportant at earlier stages of skeletonization to become important at later stages; and (2) FORCE: iterative process that allows exploration by allowing already pruned parameters to resurrect at later stages of skeletonization. Empirical analyses on a large suite of experiments show that our approach, while providing at least as good a performance as other recent approaches on moderate pruning levels, provides remarkably improved performance on higher pruning levels (could remove up to 99.5% parameters while keeping the networks trainable). Code can be found in https://github.com/naver/force.