# Towards Compute-Optimal Transfer Learning

arXiv: https://arxiv.org/abs/2304.13164 · HTML: https://mlsanity.com/paper/2304.13164/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 18 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 17 of 17)

- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers](https://mlsanity.com/paper/1906.02773/) (2019)
- [The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models](https://mlsanity.com/paper/2012.06908/) (2020)
- [Sparse Transfer Learning via Winning Lottery Tickets](https://mlsanity.com/paper/1905.07785/) (2019)
- [Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets](https://mlsanity.com/paper/2202.04736/) (2022)
- [Winning the Lottery Ahead of Time: Efficient Early Network Pruning](https://mlsanity.com/paper/2206.10451/) (2022)
- [NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research](https://mlsanity.com/paper/2211.11747/) (2022)
- [DiSparse: Disentangled Sparsification for Multitask Model Compression](https://mlsanity.com/paper/2206.04662/) (2022)
- [On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks](https://mlsanity.com/paper/2001.05050/) (2020)
- [On the Transferability of Winning Tickets in Non-Natural Image Datasets](https://mlsanity.com/paper/2005.05232/) (2020)
- [Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training](https://mlsanity.com/paper/2204.11218/) (2022)
- [Bespoke vs. Pret-`a-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding](https://mlsanity.com/paper/2007.04091/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Survey on Video Analytics in Cloud-Edge-Terminal Collaborative Systems](https://mlsanity.com/paper/2502.06581/) (2025)
- [Importance inversion transfer identifies shared principles for cross-domain learning](https://mlsanity.com/paper/2602.09116/) (2026)

## Abstract (arXiv)

The field of transfer learning is undergoing a significant shift with the introduction of large pretrained models which have demonstrated strong adaptability to a variety of downstream tasks. However, the high computational and memory requirements to finetune or use these models can be a hindrance to their widespread use. In this study, we present a solution to this issue by proposing a simple yet effective way to trade computational efficiency for asymptotic performance which we define as the performance a learning algorithm achieves as compute tends to infinity. Specifically, we argue that zero-shot structured pruning of pretrained models allows them to increase compute efficiency with minimal reduction in performance. We evaluate our method on the Nevis'22 continual learning benchmark that offers a diverse set of transfer scenarios. Our results show that pruning convolutional filters of pretrained models can lead to more than 20% performance improvement in low computational regimes.