# LogME: Practical Assessment of Pre-trained Models for Transfer Learning

arXiv: https://arxiv.org/abs/2102.11005 · HTML: https://mlsanity.com/paper/2102.11005/

## Key facts

- Cited by 172 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 77 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)

## Built on by (most cited first, 30 of 172)

- [Federated Learning from Pre-Trained Models: A Contrastive Learning Approach](https://mlsanity.com/paper/2209.10083/) (2022)
- [SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.08808/) (2022)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [A Broad Study of Pre-training for Domain Generalization and Adaptation](https://mlsanity.com/paper/2203.11819/) (2022)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [Transferability Estimation using Bhattacharyya Class Separability](https://mlsanity.com/paper/2111.12780/) (2021)
- [Pro-tuning: Unified Prompt Tuning for Vision Tasks](https://mlsanity.com/paper/2207.14381/) (2022)
- [Test-time Batch Statistics Calibration for Covariate Shift](https://mlsanity.com/paper/2110.04065/) (2021)
- [Scaling Laws for Downstream Task Performance of Large Language Models](https://mlsanity.com/paper/2402.04177/) (2024)
- [Frustratingly Easy Transferability Estimation](https://mlsanity.com/paper/2106.09362/) (2021)
- [Scalable Diverse Model Selection for Accessible Transfer Learning](https://mlsanity.com/paper/2111.06977/) (2021)
- [Zoo-Tuning: Adaptive Transfer from a Zoo of Models](https://mlsanity.com/paper/2106.15434/) (2021)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [Training-free Lexical Backdoor Attacks on Language Models](https://mlsanity.com/paper/2302.04116/) (2023)
- [PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks](https://mlsanity.com/paper/2203.05126/) (2022)
- [How stable are Transferability Metrics evaluations?](https://mlsanity.com/paper/2204.01403/) (2022)
- [Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space](https://mlsanity.com/paper/2207.03036/) (2022)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)
- [Do Multiple Instance Learning Models Transfer?](https://mlsanity.com/paper/2506.09022/) (2025)
- [Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs](https://mlsanity.com/paper/2110.10545/) (2021)
- [Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing](https://mlsanity.com/paper/2502.17282/) (2025)
- [We Should Chart an Atlas of All the World's Models](https://mlsanity.com/paper/2503.10633/) (2025)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again](https://mlsanity.com/paper/2210.04427/) (2022)
- [SphereFed: Hyperspherical Federated Learning](https://mlsanity.com/paper/2207.09413/) (2022)
- [How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?](https://mlsanity.com/paper/2501.11253/) (2025)
- [Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How](https://mlsanity.com/paper/2306.03828/) (2023)
- [ETran: Energy-Based Transferability Estimation](https://mlsanity.com/paper/2308.02027/) (2023)
- [The Role of Pre-training Data in Transfer Learning](https://mlsanity.com/paper/2302.13602/) (2023)

## Abstract (arXiv)

This paper studies task adaptive pre-trained model selection, an underexplored problem of assessing pre-trained models for the target task and select best ones from the model zoo without fine-tuning. A few pilot works addressed the problem in transferring supervised pre-trained models to classification tasks, but they cannot handle emerging unsupervised pre-trained models or regression tasks. In pursuit of a practical assessment method, we propose to estimate the maximum value of label evidence given features extracted by pre-trained models. Unlike the maximum likelihood, the maximum evidence is immune to over-fitting, while its expensive computation can be dramatically reduced by our carefully designed algorithm. The Logarithm of Maximum Evidence (LogME) can be used to assess pre-trained models for transfer learning: a pre-trained model with a high LogME value is likely to have good transfer performance. LogME is fast, accurate, and general, characterizing itself as the first practical method for assessing pre-trained models. Compared with brute-force fine-tuning, LogME brings at most 3000× speedup in wall-clock time and requires only 1% memory footprint. It outperforms prior methods by a large margin in their setting and is applicable to new settings. It is general enough for diverse pre-trained models (supervised pre-trained and unsupervised pre-trained), downstream tasks (classification and regression), and modalities (vision and language). Code is available at this repository: hrefhttps://github.com/thuml/LogMEhttps://github.com/thuml/LogME.