mlsanityData to 14 Oct 2026

Scalable Diverse Model Selection for Accessible Transfer Learning

arXiv 2111.06977 ↗First posted 12 Nov 2021cs.LG cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

With the preponderance of pretrained deep learning models available off-the-shelf from model banks today, finding the best weights to fine-tune to your use-case can be a daunting task. Several methods have recently been proposed to find good models for transfer learning, but they either don't scale well to large model banks or don't perform well on the diversity of off-the-shelf models. Ideally the question we want to answer is, "given some data and a source model, can you quickly predict the model's accuracy after fine-tuning?" In this paper, we formalize this setting as "Scalable Diverse Model Selection" and propose several benchmarks for evaluating on this task. We find that existing model selection and transferability estimation methods perform poorly here and analyze why this is the case. We then introduce simple techniques to improve the performance and speed of these algorithms. Finally, we iterate on existing methods to create PARC, which outperforms all other methods on diverse model selection. We have released the benchmarks and method code in hope to inspire future work in model selection for accessible transfer learning.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 12 Nov 2021.

Builds on

41 of 62 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2017Mask R-CNNcited by 7,277
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  10. 2014Going Deeper with Convolutionscited by 4,241
  11. 2018YOLOv3: An Incremental Improvementcited by 2,659
  12. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
29 more
  1. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  3. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  4. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  5. 2019Panoptic Feature Pyramid Networkscited by 645
  6. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  7. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  8. 2018Do Better ImageNet Models Transfer Better?cited by 565
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  10. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  11. 2018A Survey on Deep Transfer Learningcited by 419
  12. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  13. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  14. 2016What makes ImageNet good for transfer learning?cited by 192
  15. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 188
  16. 2018Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learningcited by 174
  17. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  18. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
  19. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  20. 2019Transferability and Hardness of Supervised Classification Taskscited by 127
  21. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  22. 2020Predicting Neural Network Accuracy from Weightscited by 110
  23. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  24. 2020Scalable Transfer Learning with Expert Modelscited by 49
  25. 2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
  26. 2014Analyzing the Performance of Multilayer Neural Networks for Object Recognitioncited by 29
  27. 2019Deep Model Transferability from Attribution Mapscited by 28
  28. 2020Neural Data Server: A Large-Scale Search Engine for Transfer Learning Datacited by 27
  29. 2020Duality Diagram Similarity: a generic framework for initialization selection in task transfer learningcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 10; 2024: 10; 2025: 9; 2026: 4.

40 papers.

Sort:

By area:

  1. 2022Deep Model Reassemblycited by 78
  2. 2023Model Spider: Learning to Rank Pre-Trained Models Efficientlycited by 32
  3. 2022How stable are Transferability Metrics evaluations?cited by 30
  4. 2023Scalable Optimal Transport Methods in Machine Learning: A Contemporary Surveycited by 27
  5. 2021Transferability Metrics for Selecting Source Model Ensemblescited by 24
  6. 2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 18
  7. 2023The Role of Pre-training Data in Transfer Learningcited by 17
  8. 2022Deep transfer learning for image classification: a surveycited by 13
  9. 2023Exploring Model Transferability through the Lens of Potential Energycited by 11
  10. 2021Graph-Based Similarity of Neural Network Representationscited by 7
  11. 2022Content-Based Search for Deep Generative Modelscited by 7
  12. 2024Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Componentscited by 6
28 more
  1. 2022Hub-Pathway: Transfer Learning from A Hub of Pre-trained Modelscited by 5
  2. 2023Simple Transferability Estimation for Regression Taskscited by 5
  3. 2024Model Selection with Model Zoo via Graph Learningcited by 5
  4. 2025Exploring Structural Degradation in Dense Representations for Self-supervised Learningcited by 5
  5. 2023Distilling from Similar Tasks for Transfer Learning on a Budgetcited by 4
  6. 2023Efficient Stitchable Task Adaptationcited by 4
  7. 2023How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Surveycited by 4
  8. 2024Labeled Data Selection for Category Discoverycited by 4
  9. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distancecited by 4
  10. 2025LEAD: Exploring Logit Space Evolution for Model Selectioncited by 3
  11. 2023Unleashing the power of Neural Collapse for Transferability Estimationcited by 1
  12. 2024Selection, Ensemble, and Adaptation: Advancing Multi-Source-Free Domain Adaptation via Architecture Zoocited by 1
  13. 2024Pre-Trained Model Recommendation for Downstream Fine-tuningcited by 1
  14. 2024KITE: A Kernel-based Improved Transferability Estimation Methodcited by 1
  15. 2024Towards Fundamentally Scalable Model Selection: Asymptotically Fast Update and Selectioncited by 1
  16. 2025k-NN as a Simple and Effective Estimator of Transferabilitycited by 1
  17. 2025A Contrastive Diffusion-based Network (CDNet) for Time Series Classificationcited by 1
  18. 2025Match & Choose: Model Selection Framework for Fine-tuning Text-to-Image Diffusion Modelscited by 1
  19. 2026Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Modelscited by 1
  20. 2024Green Runner: A tool for efficient deep learning component selectioncited by 0
  21. 2024Flatness Improves Backbone Generalisation in Few-shot Classificationcited by 0
  22. 2024Back to the Basics on Predicting Transfer Performancecited by 0
  23. 2025How to Select Pre-Trained Code Models for Reuse? A Learning Perspectivecited by 0
  24. 2025Implicit Modeling for Transferability Estimation of Vision Foundation Modelscited by 0
  25. 2025MetaRank: Task-Aware Metric Selection for Model Transferability Estimationcited by 0
  26. 2025Neural Coherence : Find higher performance to out-of-distribution tasks from few samplescited by 0
  27. 2026One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensemblingcited by 0
  28. 2026Robustness of transferability estimation metrics for medical imagingcited by 0