# Single Shot Structured Pruning Before Training

arXiv: https://arxiv.org/abs/2007.00389 · HTML: https://mlsanity.com/paper/2007.00389/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2020.

## Built on by (most cited first, 19 of 19)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Advancing Model Pruning via Bi-level Optimization](https://mlsanity.com/paper/2210.04092/) (2022)
- [Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients](https://mlsanity.com/paper/2202.08132/) (2022)
- [SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning](https://mlsanity.com/paper/2110.11395/) (2021)
- [Effective Model Sparsification by Scheduled Grow-and-Prune Methods](https://mlsanity.com/paper/2106.09857/) (2021)
- [Differentiable Network Pruning for Microcontrollers](https://mlsanity.com/paper/2110.08350/) (2021)
- [Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training](https://mlsanity.com/paper/2209.11204/) (2022)
- [EPSD: Early Pruning with Self-Distillation for Efficient Model Compression](https://mlsanity.com/paper/2402.00084/) (2024)
- [Sparse then Prune: Toward Efficient Vision Transformers](https://mlsanity.com/paper/2307.11988/) (2023)
- [AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networks](https://mlsanity.com/paper/2304.06941/) (2023)
- [Pruning at Initialization -- A Sketching Perspective](https://mlsanity.com/paper/2305.17559/) (2023)
- [Pruning-based Data Selection and Network Fusion for Efficient Deep Learning](https://mlsanity.com/paper/2501.01118/) (2025)
- [Rapid Deployment of DNNs for Edge Computing via Structured Pruning at Initialization](https://mlsanity.com/paper/2404.16877/) (2024)
- [LinDeps: A Fine-tuning Free Post-Pruning Method to Remove Layer-Wise Linear Dependencies with Guaranteed Performance Preservation](https://mlsanity.com/paper/2507.21573/) (2025)
- [Hyperparameter Optimization with Neural Network Pruning](https://mlsanity.com/paper/2205.08695/) (2022)
- [Exploring Vision Neural Network Pruning via Screening Methodology](https://mlsanity.com/paper/2502.07189/) (2025)
- [PruneFuse: Efficient Data Selection via Weight Pruning and Network Fusion](https://mlsanity.com/paper/2603.26138/) (2026)
- [FedASAP: Activation Statistics-driven Structured Adaptive Pruning for Efficient Personalized Federated Learning for Lesion Segmentation on brain MRI](https://mlsanity.com/paper/2609.19042/) (2026)

## Abstract (arXiv)

We introduce a method to speed up training by 2x and inference by 3x in deep neural networks using structured pruning applied before training. Unlike previous works on pruning before training which prune individual weights, our work develops a methodology to remove entire channels and hidden units with the explicit aim of speeding up training and inference. We introduce a compute-aware scoring mechanism which enables pruning in units of sensitivity per FLOP removed, allowing even greater speed ups. Our method is fast, easy to implement, and needs just one forward/backward pass on a single batch of data to complete pruning before training begins.