# Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients

arXiv: https://arxiv.org/abs/2202.08132 · HTML: https://mlsanity.com/paper/2202.08132/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis](https://mlsanity.com/paper/1905.05934/) (2019)
- [Progressive Skeletonization: Trimming more fat from a network at initialization](https://mlsanity.com/paper/2006.09081/) (2020)
- [Single Shot Structured Pruning Before Training](https://mlsanity.com/paper/2007.00389/) (2020)
- [Semi-Supervised Learning with Meta-Gradient](https://mlsanity.com/paper/2007.03966/) (2020)

## Built on by (most cited first, 30 of 34)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Advancing Model Pruning via Bi-level Optimization](https://mlsanity.com/paper/2210.04092/) (2022)
- [Rare Gems: Finding Lottery Tickets at Initialization](https://mlsanity.com/paper/2202.12002/) (2022)
- [NTK-SAP: Improving neural network pruning by aligning training dynamics](https://mlsanity.com/paper/2304.02840/) (2023)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning](https://mlsanity.com/paper/2404.05621/) (2024)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [A Unified Framework for Soft Threshold Pruning](https://mlsanity.com/paper/2302.13019/) (2023)
- [Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruning](https://mlsanity.com/paper/2406.01820/) (2024)
- [Class-Aware Pruning for Efficient Neural Networks](https://mlsanity.com/paper/2312.05875/) (2023)
- [Balanced Training for Sparse GANs](https://mlsanity.com/paper/2302.14670/) (2023)
- [DNNShifter: An Efficient DNN Pruning System for Edge Computing](https://mlsanity.com/paper/2309.06973/) (2023)
- [No Free Prune: Information-Theoretic Barriers to Pruning at Initialization](https://mlsanity.com/paper/2402.01089/) (2024)
- [The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models](https://mlsanity.com/paper/2507.18725/) (2025)
- [EPSD: Early Pruning with Self-Distillation for Efficient Model Compression](https://mlsanity.com/paper/2402.00084/) (2024)
- [Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch](https://mlsanity.com/paper/2504.12801/) (2025)
- [Magnitude Attention-based Dynamic Pruning](https://mlsanity.com/paper/2306.05056/) (2023)
- [In defense of parameter sharing for model-compression](https://mlsanity.com/paper/2310.11611/) (2023)
- [Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks](https://mlsanity.com/paper/2205.15619/) (2022)
- [One-shot Network Pruning at Initialization with Discriminative Image Patches](https://mlsanity.com/paper/2209.05683/) (2022)
- [Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations](https://mlsanity.com/paper/2305.16886/) (2023)
- [Pruning at Initialization -- A Sketching Perspective](https://mlsanity.com/paper/2305.17559/) (2023)
- [The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis](https://mlsanity.com/paper/2510.17515/) (2025)
- [NeuroFlux: Memory-Efficient CNN Training Using Adaptive Local Learning](https://mlsanity.com/paper/2402.14139/) (2024)
- [Rapid Deployment of DNNs for Edge Computing via Structured Pruning at Initialization](https://mlsanity.com/paper/2404.16877/) (2024)
- [AutoSculpt: A Pattern-based Model Auto-pruning Framework Using Reinforcement Learning and Graph Learning](https://mlsanity.com/paper/2412.18091/) (2024)
- [Safe Vision-Language Models via Unsafe Weights Manipulation](https://mlsanity.com/paper/2503.11742/) (2025)
- [SAP: Syntactic Attention Pruning for Transformer-based Language Models](https://mlsanity.com/paper/2512.19125/) (2025)
- [Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models](https://mlsanity.com/paper/2608.06901/) (2026)

## Abstract (arXiv)

Pruning neural networks at initialization would enable us to find sparse models that retain the accuracy of the original network while consuming fewer computational resources for training and inference. However, current methods are insufficient to enable this optimization and lead to a large degradation in model performance. In this paper, we identify a fundamental limitation in the formulation of current methods, namely that their saliency criteria look at a single step at the start of training without taking into account the trainability of the network. While pruning iteratively and gradually has been shown to improve pruning performance, explicit consideration of the training stage that will immediately follow pruning has so far been absent from the computation of the saliency criterion. To overcome the short-sightedness of existing methods, we propose Prospect Pruning (ProsPr), which uses meta-gradients through the first few steps of optimization to determine which weights to prune. ProsPr combines an estimate of the higher-order effects of pruning on the loss and the optimization trajectory to identify the trainable sub-network. Our method achieves state-of-the-art pruning performance on a variety of vision classification tasks, with less data and in a single shot compared to existing pruning-at-initialization methods.