mlsanityData to 7 Oct 2026

PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks

arXiv 2203.05126 ↗First posted 10 Mar 2022cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

With the increasing abundance of pretrained models in recent years, the problem of selecting the best pretrained checkpoint for a particular downstream classification task has been gaining increased attention. Although several methods have recently been proposed to tackle the selection problem (e.g. LEEP, H-score), these methods resort to applying heuristics that are not well motivated by learning theory. In this paper we present PACTran, a theoretically grounded family of metrics for pretrained model selection and transferability measurement. We first show how to derive PACTran metrics from the optimal PAC-Bayesian bound under the transfer learning setting. We then empirically evaluate three metric instantiations of PACTran on a number of vision tasks (VTAB) as well as a language-and-vision (OKVQA) task. An analysis of the results shows PACTran is a more consistent and effective transferability measure compared to existing selection methods.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 41 of its 77 references that are in the map. First posted on arXiv 10 Mar 2022.

Builds on

41 of 77 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,960
29 more
  1. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  2. 2016Variational Inference: A Review for Statisticianscited by 1,466
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  9. 2017Exploring Generalization in Deep Learningcited by 595
  10. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  11. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  12. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  13. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  14. 2017Wasserstein Auto-Encoderscited by 371
  15. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  16. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  17. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  18. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  19. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
  20. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  21. 2015Exploring Models and Data for Image Question Answeringcited by 139
  22. 2019Transferability and Hardness of Supervised Classification Taskscited by 125
  23. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  24. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  25. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 58
  26. 2020Ranking Neural Checkpointscited by 50
  27. 2019Practical and Consistent Estimation of f-Divergencescited by 24
  28. 2021Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learningcited by 21
  29. 2019On Universal Features for High-Dimensional Learning and Inferencecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 7; 2025: 7; 2026: 5.

30 papers.

Sort:

By area:

  1. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  2. 2023Demystifying Structural Disparity in Graph Neural Networks: Can One Size Fit All?cited by 41
  3. 2023Model Spider: Learning to Rank Pre-Trained Models Efficientlycited by 32
  4. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  5. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  6. 2023ETran: Energy-Based Transferability Estimationcited by 18
  7. 2023Foundation Model is Efficient Multimodal Multitask Model Selectorcited by 16
  8. 2024Which Model to Transfer? A Survey on Transferability Estimationcited by 13
  9. 2023LOVM: Language-Only Vision Model Selectioncited by 12
  10. 2023Understanding the Transferability of Representations via Task-Relatednesscited by 11
  11. 2023Exploring Model Transferability through the Lens of Potential Energycited by 10
  12. 2022Improving Robust Generalization by Direct PAC-Bayesian Bound Minimizationcited by 6
18 more
  1. 2024Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Componentscited by 6
  2. 2023How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Surveycited by 4
  3. 2025Many-Task Federated Fine-Tuning via Unified Task Vectorscited by 3
  4. 2025LEAD: Exploring Logit Space Evolution for Model Selectioncited by 3
  5. 2025Benchmarking Transferability: A Framework for Fair and Robust Evaluationcited by 2
  6. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distancecited by 2
  7. 2023To transfer or not transfer: Unified transferability metric and analysiscited by 1
  8. 2024When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspectivecited by 1
  9. 2025k-NN as a Simple and Effective Estimator of Transferabilitycited by 1
  10. 2026ModelLens: Finding the Best for Your Task from Myriads of Modelscited by 1
  11. 2024Green Runner: A tool for efficient deep learning component selectioncited by 0
  12. 2024Efficient Transferability Assessment for Selection of Pre-trained Detectorscited by 0
  13. 2024Back to the Basics on Predicting Transfer Performancecited by 0
  14. 2024Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Rankingcited by 0
  15. 2025Implicit Modeling for Transferability Estimation of Vision Foundation Modelscited by 0
  16. 2026Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentationcited by 0
  17. 2026Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Modelscited by 0
  18. 2026When More Foundation Models Means Less: Diagnosing and Addressing Multi-View Fusion Failurecited by 0