# Pruning from Scratch

arXiv: https://arxiv.org/abs/1909.12579 · HTML: https://mlsanity.com/paper/1909.12579/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2019.

## Built on by (most cited first, 30 of 72)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance](https://mlsanity.com/paper/2309.12314/) (2023)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [Combined Depth Space based Architecture Search For Person Re-identification](https://mlsanity.com/paper/2104.04163/) (2021)
- [Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2407.09590/) (2024)
- [CHEX: CHannel EXploration for CNN Model Compression](https://mlsanity.com/paper/2203.15794/) (2022)
- [CondenseNet V2: Sparse Feature Reactivation for Deep Networks](https://mlsanity.com/paper/2104.04382/) (2021)
- [GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization](https://mlsanity.com/paper/2109.02220/) (2021)
- [Locally Free Weight Sharing for Network Width Search](https://mlsanity.com/paper/2102.05258/) (2021)
- [Pruning for Robust Concept Erasing in Diffusion Models](https://mlsanity.com/paper/2405.16534/) (2024)
- [Pruning Randomly Initialized Neural Networks with Iterative Randomization](https://mlsanity.com/paper/2106.09269/) (2021)
- [Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions](https://mlsanity.com/paper/2209.08554/) (2022)
- [ModuleFormer: Modularity Emerges from Mixture-of-Experts](https://mlsanity.com/paper/2306.04640/) (2023)
- [Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis](https://mlsanity.com/paper/2104.04868/) (2021)
- [SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks](https://mlsanity.com/paper/2102.03773/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Complement Sparsification: Low-Overhead Model Pruning for Federated Learning](https://mlsanity.com/paper/2303.06237/) (2023)
- [Automatic Attention Pruning: Improving and Automating Model Pruning using Attentions](https://mlsanity.com/paper/2303.08595/) (2023)
- [Lottery Jackpots Exist in Pre-trained Models](https://mlsanity.com/paper/2104.08700/) (2021)
- [An Effective Information Theoretic Framework for Channel Pruning](https://mlsanity.com/paper/2408.16772/) (2024)
- [EZCrop: Energy-Zoned Channels for Robust Output Pruning](https://mlsanity.com/paper/2105.03679/) (2021)
- [CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization](https://mlsanity.com/paper/2110.10921/) (2021)
- [Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks](https://mlsanity.com/paper/2211.12714/) (2022)
- [Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks](https://mlsanity.com/paper/2310.19470/) (2023)
- [Quantum Ridgelet Transform: Winning Lottery Ticket of Neural Networks with Quantum Computation](https://mlsanity.com/paper/2301.11936/) (2023)
- [CFDP: Common Frequency Domain Pruning](https://mlsanity.com/paper/2306.04147/) (2023)
- [BYOM: Building Your Own Multi-Task Model For Free](https://mlsanity.com/paper/2310.01886/) (2023)
- [Deeplite Neutrino: An End-to-End Framework for Constrained Deep Learning Model Optimization](https://mlsanity.com/paper/2101.04073/) (2021)
- [AACP: Model Compression by Accurate and Automatic Channel Pruning](https://mlsanity.com/paper/2102.00390/) (2021)
- [Finding Everything within Random Binary Networks](https://mlsanity.com/paper/2110.08996/) (2021)

## Abstract (arXiv)

Network pruning is an important research field aiming at reducing computational costs of neural networks. Conventional approaches follow a fixed paradigm which first trains a large and redundant network, and then determines which units (e.g., channels) are less important and thus can be removed. In this work, we find that pre-training an over-parameterized model is not necessary for obtaining the target pruned structure. In fact, a fully-trained over-parameterized model will reduce the search space for the pruned structure. We empirically show that more diverse pruned structures can be directly pruned from randomly initialized weights, including potential models with better performance. Therefore, we propose a novel network pruning pipeline which allows pruning from scratch. In the experiments for compressing classification models on CIFAR10 and ImageNet datasets, our approach not only greatly reduces the pre-training burden of traditional pruning methods, but also achieves similar or even higher accuracy under the same computation budgets. Our results facilitate the community to rethink the effectiveness of existing techniques used for network pruning.