# How stable are Transferability Metrics evaluations?

arXiv: https://arxiv.org/abs/2204.01403 · HTML: https://mlsanity.com/paper/2204.01403/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 98 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 30 of 67)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)

## Built on by (most cited first, 30 of 30)

- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [Can We Evaluate Domain Adaptation Models Without Target-Domain Labels?](https://mlsanity.com/paper/2305.18712/) (2023)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [Which Model to Transfer? A Survey on Transferability Estimation](https://mlsanity.com/paper/2402.15231/) (2024)
- [LOVM: Language-Only Vision Model Selection](https://mlsanity.com/paper/2306.08893/) (2023)
- [A Bag-of-Prototypes Representation for Dataset-Level Applications](https://mlsanity.com/paper/2303.13251/) (2023)
- [Exploring Model Transferability through the Lens of Potential Energy](https://mlsanity.com/paper/2308.15074/) (2023)
- [How to Estimate Model Transferability of Pre-Trained Speech Models?](https://mlsanity.com/paper/2306.01015/) (2023)
- [Better Practices for Domain Adaptation](https://mlsanity.com/paper/2309.03879/) (2023)
- [The Performance of Transferability Metrics does not Translate to Medical Tasks](https://mlsanity.com/paper/2308.07444/) (2023)
- [Sequence Transferability and Task Order Selection in Continual Learning](https://mlsanity.com/paper/2502.06544/) (2025)
- [How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey](https://mlsanity.com/paper/2312.04775/) (2023)
- [Understanding Knowledge Transferability for Transfer Learning: A Survey](https://mlsanity.com/paper/2507.03175/) (2025)
- [Sensitivity analysis of AI-based algorithms for autonomous driving on optical wavefront aberrations induced by the windshield](https://mlsanity.com/paper/2308.11711/) (2023)
- [To transfer or not transfer: Unified transferability metric and analysis](https://mlsanity.com/paper/2305.07741/) (2023)
- [Towards Fundamentally Scalable Model Selection: Asymptotically Fast Update and Selection](https://mlsanity.com/paper/2406.07536/) (2024)
- [k-NN as a Simple and Effective Estimator of Transferability](https://mlsanity.com/paper/2503.18528/) (2025)
- [Towards Estimating Transferability using Hard Subsets](https://mlsanity.com/paper/2301.06928/) (2023)
- [Efficient Transferability Assessment for Selection of Pre-trained Detectors](https://mlsanity.com/paper/2403.09432/) (2024)
- [Back to the Basics on Predicting Transfer Performance](https://mlsanity.com/paper/2405.20420/) (2024)
- [Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking](https://mlsanity.com/paper/2409.16198/) (2024)
- [Enhancing pretraining efficiency for medical image segmentation via transferability metrics](https://mlsanity.com/paper/2410.18677/) (2024)
- [How to Select Pre-Trained Code Models for Reuse? A Learning Perspective](https://mlsanity.com/paper/2501.03783/) (2025)
- [Unsupervised Source-Free Ranking of Biomedical Segmentation Models Under Distribution Shift](https://mlsanity.com/paper/2503.00450/) (2025)
- [PEARL: Performance-Enhanced Aggregated Representation Learning](https://mlsanity.com/paper/2509.24312/) (2025)
- [Implicit Modeling for Transferability Estimation of Vision Foundation Models](https://mlsanity.com/paper/2510.23145/) (2025)
- [Efficient, Validation-Free Intrinsic Quality Estimation for Large-Scale Face Recognition Datasets](https://mlsanity.com/paper/2605.29720/) (2026)
- [One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensembling](https://mlsanity.com/paper/2606.08126/) (2026)
- [Strategy, Not Payoffs: A Behavioural Embedding of Normal-Form Games](https://mlsanity.com/paper/2607.27536/) (2026)
- [Robustness of transferability estimation metrics for medical imaging](https://mlsanity.com/paper/2608.09999/) (2026)

## Abstract (arXiv)

Transferability metrics is a maturing field with increasing interest, which aims at providing heuristics for selecting the most suitable source models to transfer to a given target dataset, without fine-tuning them all. However, existing works rely on custom experimental setups which differ across papers, leading to inconsistent conclusions about which transferability metrics work best. In this paper we conduct a large-scale study by systematically constructing a broad range of 715k experimental setup variations. We discover that even small variations to an experimental setup lead to different conclusions about the superiority of a transferability metric over another. Then we propose better evaluations by aggregating across many experiments, enabling to reach more stable conclusions. As a result, we reveal the superiority of LogME at selecting good source datasets to transfer from in a semantic segmentation scenario, NLEEP at selecting good source architectures in an image classification scenario, and GBC at determining which target task benefits most from a given source model. Yet, no single transferability metric works best in all scenarios.