mlsanityData to 7 Oct 2026

How stable are Transferability Metrics evaluations?

arXiv 2204.01403 ↗First posted 4 Apr 2022cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

Transferability metrics is a maturing field with increasing interest, which aims at providing heuristics for selecting the most suitable source models to transfer to a given target dataset, without fine-tuning them all. However, existing works rely on custom experimental setups which differ across papers, leading to inconsistent conclusions about which transferability metrics work best. In this paper we conduct a large-scale study by systematically constructing a broad range of 715k experimental setup variations. We discover that even small variations to an experimental setup lead to different conclusions about the superiority of a transferability metric over another. Then we propose better evaluations by aggregating across many experiments, enabling to reach more stable conclusions. As a result, we reveal the superiority of LogME at selecting good source datasets to transfer from in a semantic segmentation scenario, NLEEP at selecting good source architectures in an image classification scenario, and GBC at determining which target task benefits most from a given source model. Yet, no single transferability metric works best in all scenarios.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 67 of its 98 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

67 of 98 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2017Mask R-CNNcited by 7,259
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
38 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2015Fast R-CNNcited by 2,930
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2019Searching for MobileNetV3cited by 1,984
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  11. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  12. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  13. 2019Objects as Pointscited by 1,172
  14. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  15. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  16. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  17. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  18. 2018Panoptic Segmentationcited by 796
  19. 2020Benchmarking Graph Neural Networkscited by 672
  20. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  21. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 622
  22. 2021ImageNet-21K Pretraining for the Massescited by 607
  23. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  24. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  25. 2020Virtual KITTI 2cited by 345
  26. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 340
  27. 2019A Fair Comparison of Graph Neural Networks for Graph Classificationcited by 244
  28. 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 195
  29. 2016What makes ImageNet good for transfer learning?cited by 192
  30. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
  31. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  32. 2018IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environmentscited by 163
  33. 2020Geometric Dataset Distances via Optimal Transportcited by 157
  34. 2019Transferability and Hardness of Supervised Classification Taskscited by 125
  35. 2017Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenescited by 122
  36. 2021MSeg: A Composite Dataset for Multi-domain Semantic Segmentationcited by 122
  37. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  38. 2020Semantic Segmentation of Underwater Imagery: Dataset and Benchmarkcited by 95

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 6; 2025: 7; 2026: 4.

30 papers.

Sort:

By area:

  1. 2023Model Spider: Learning to Rank Pre-Trained Models Efficientlycited by 32
  2. 2023Can We Evaluate Domain Adaptation Models Without Target-Domain Labels?cited by 14
  3. 2022Deep transfer learning for image classification: a surveycited by 13
  4. 2024Which Model to Transfer? A Survey on Transferability Estimationcited by 13
  5. 2023LOVM: Language-Only Vision Model Selectioncited by 12
  6. 2023A Bag-of-Prototypes Representation for Dataset-Level Applicationscited by 11
  7. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  8. 2023Exploring Model Transferability through the Lens of Potential Energycited by 10
  9. 2023Better Practices for Domain Adaptationcited by 9
  10. 2023The Performance of Transferability Metrics does not Translate to Medical Taskscited by 6
  11. 2025Sequence Transferability and Task Order Selection in Continual Learningcited by 5
  12. 2023How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Surveycited by 4
18 more
  1. 2025Understanding Knowledge Transferability for Transfer Learning: A Surveycited by 4
  2. 2023Sensitivity analysis of AI-based algorithms for autonomous driving on optical wavefront aberrations induced by the windshieldcited by 2
  3. 2023To transfer or not transfer: Unified transferability metric and analysiscited by 1
  4. 2024Towards Fundamentally Scalable Model Selection: Asymptotically Fast Update and Selectioncited by 1
  5. 2025k-NN as a Simple and Effective Estimator of Transferabilitycited by 1
  6. 2023Towards Estimating Transferability using Hard Subsetscited by 0
  7. 2024Efficient Transferability Assessment for Selection of Pre-trained Detectorscited by 0
  8. 2024Back to the Basics on Predicting Transfer Performancecited by 0
  9. 2024Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Rankingcited by 0
  10. 2024Enhancing pretraining efficiency for medical image segmentation via transferability metricscited by 0
  11. 2025How to Select Pre-Trained Code Models for Reuse? A Learning Perspectivecited by 0
  12. 2025Unsupervised Source-Free Ranking of Biomedical Segmentation Models Under Distribution Shiftcited by 0
  13. 2025PEARL: Performance-Enhanced Aggregated Representation Learningcited by 0
  14. 2025Implicit Modeling for Transferability Estimation of Vision Foundation Modelscited by 0
  15. 2026Efficient, Validation-Free Intrinsic Quality Estimation for Large-Scale Face Recognition Datasetscited by 0
  16. 2026One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensemblingcited by 0
  17. 2026Strategy, Not Payoffs: A Behavioural Embedding of Normal-Form Gamescited by 0
  18. 2026Robustness of transferability estimation metrics for medical imagingcited by 0