# Which Model to Transfer? Finding the Needle in the Growing Haystack

arXiv: https://arxiv.org/abs/2010.06402 · HTML: https://mlsanity.com/paper/2010.06402/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [Self-Supervised Pretraining Improves Self-Supervised Pretraining](https://mlsanity.com/paper/2103.12718/) (2021)
- [What to Pre-Train on? Efficient Intermediate Task Selection](https://mlsanity.com/paper/2104.08247/) (2021)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [Do Multiple Instance Learning Models Transfer?](https://mlsanity.com/paper/2506.09022/) (2025)
- [Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreement](https://mlsanity.com/paper/2305.01481/) (2023)
- [SHiFT: An Efficient, Flexible Search Engine for Transfer Learning](https://mlsanity.com/paper/2204.01457/) (2022)
- [Transferability Estimation Based On Principal Gradient Expectation](https://mlsanity.com/paper/2211.16299/) (2022)
- [Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning](https://mlsanity.com/paper/2407.16245/) (2024)
- [Beyond Knowledge Silos: Task Fingerprinting for Democratization of Medical Imaging AI](https://mlsanity.com/paper/2412.08763/) (2024)
- [Selective Cross-Task Distillation](https://mlsanity.com/paper/2204.11526/) (2022)
- [Green Runner: A tool for efficient model selection from model repositories](https://mlsanity.com/paper/2305.16849/) (2023)
- [Leveraging The Edge-to-Cloud Continuum for Scalable Machine Learning on Decentralized Data](https://mlsanity.com/paper/2306.10848/) (2023)
- [k-NN as a Simple and Effective Estimator of Transferability](https://mlsanity.com/paper/2503.18528/) (2025)
- [Learning Downstream Task by Selectively Capturing Complementary Knowledge from Multiple Self-supervisedly Learning Pretexts](https://mlsanity.com/paper/2204.05248/) (2022)
- [A Two-Phase Recall-and-Select Framework for Fast Model Selection](https://mlsanity.com/paper/2404.00069/) (2024)
- [On Supernet Transfer Learning for Effective Task Adaptation](https://mlsanity.com/paper/2407.20279/) (2024)
- [How to Select Pre-Trained Code Models for Reuse? A Learning Perspective](https://mlsanity.com/paper/2501.03783/) (2025)
- [Unsupervised Source-Free Ranking of Biomedical Segmentation Models Under Distribution Shift](https://mlsanity.com/paper/2503.00450/) (2025)
- [Mitigating Pretraining-Induced Attention Asymmetry in 2D+ Electron Microscopy Image Segmentation](https://mlsanity.com/paper/2505.14105/) (2025)
- [Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement](https://mlsanity.com/paper/2512.05525/) (2025)
- [LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations](https://mlsanity.com/paper/2609.28086/) (2026)

## Abstract (arXiv)

Transfer learning has been recently popularized as a data-efficient alternative to training models from scratch, in particular for computer vision tasks where it provides a remarkably solid baseline. The emergence of rich model repositories, such as TensorFlow Hub, enables the practitioners and researchers to unleash the potential of these models across a wide range of downstream tasks. As these repositories keep growing exponentially, efficiently selecting a good model for the task at hand becomes paramount. We provide a formalization of this problem through a familiar notion of regret and introduce the predominant strategies, namely task-agnostic (e.g. ranking models by their ImageNet performance) and task-aware search strategies (such as linear or kNN evaluation). We conduct a large-scale empirical study and show that both task-agnostic and task-aware methods can yield high regret. We then propose a simple and computationally efficient hybrid search strategy which outperforms the existing approaches. We highlight the practical benefits of the proposed solution on a set of 19 diverse vision tasks.