# One-Shot Pruning for Fast-adapting Pre-trained Models on Devices

arXiv: https://arxiv.org/abs/2307.04365 · HTML: https://mlsanity.com/paper/2307.04365/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures](https://mlsanity.com/paper/1607.03250/) (2016)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [Patch Slimming for Efficient Vision Transformers](https://mlsanity.com/paper/2106.02852/) (2021)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [Winning the Lottery with Continuous Sparsification](https://mlsanity.com/paper/1912.04427/) (2019)
- [Vision Transformer Pruning](https://mlsanity.com/paper/2104.08500/) (2021)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Unified Visual Transformer Compression](https://mlsanity.com/paper/2203.08243/) (2022)
- [Dual Lottery Ticket Hypothesis](https://mlsanity.com/paper/2203.04248/) (2022)
- [From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression](https://mlsanity.com/paper/2112.07198/) (2021)
- [Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads](https://mlsanity.com/paper/2011.03770/) (2020)
- [Meta-Learning with Network Pruning](https://mlsanity.com/paper/2007.03219/) (2020)
- [Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough](https://mlsanity.com/paper/2010.15969/) (2020)

## Built on by (most cited first, 1 of 1)

- [DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices](https://mlsanity.com/paper/2412.05823/) (2024)

## Abstract (arXiv)

Large-scale pre-trained models have been remarkably successful in resolving downstream tasks. Nonetheless, deploying these models on low-capability devices still requires an effective approach, such as model pruning. However, pruning the model from scratch can pose a practical challenge given the limited resources of each downstream task or device. To tackle this issue, we present a scalable one-shot pruning method that leverages pruned knowledge of similar tasks to extract a sub-network from the pre-trained model for a new task. Specifically, we create a score mask using the pruned models of similar tasks to identify task-specific filters/nodes in the pre-trained model for the new task. Based on this mask, we conduct a single round of pruning to extract a suitably-sized sub-network that can quickly adapt to the new task with only a few training iterations. Our experimental analysis demonstrates the effectiveness of the proposed method on the convolutional neural networks (CNNs) and vision transformers (ViT) with various datasets. The proposed method consistently outperforms popular pruning baseline methods in terms of accuracy and efficiency when dealing with diverse downstream tasks with different memory constraints.