# Ranking Neural Checkpoints

arXiv: https://arxiv.org/abs/2011.11200 · HTML: https://mlsanity.com/paper/2011.11200/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 30 of 51)

- [What to Pre-Train on? Efficient Intermediate Task Selection](https://mlsanity.com/paper/2104.08247/) (2021)
- [Transferability Estimation using Bhattacharyya Class Separability](https://mlsanity.com/paper/2111.12780/) (2021)
- [Frustratingly Easy Transferability Estimation](https://mlsanity.com/paper/2106.09362/) (2021)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks](https://mlsanity.com/paper/2203.05126/) (2022)
- [How stable are Transferability Metrics evaluations?](https://mlsanity.com/paper/2204.01403/) (2022)
- [Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space](https://mlsanity.com/paper/2207.03036/) (2022)
- [Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing](https://mlsanity.com/paper/2502.17282/) (2025)
- [We Should Chart an Atlas of All the World's Models](https://mlsanity.com/paper/2503.10633/) (2025)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [ETran: Energy-Based Transferability Estimation](https://mlsanity.com/paper/2308.02027/) (2023)
- [Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance](https://mlsanity.com/paper/2110.06893/) (2021)
- [Foundation Model is Efficient Multimodal Multitask Model Selector](https://mlsanity.com/paper/2308.06262/) (2023)
- [Which Model to Transfer? A Survey on Transferability Estimation](https://mlsanity.com/paper/2402.15231/) (2024)
- [Pick the Best Pre-trained Model: Towards Transferability Estimation for Medical Image Segmentation](https://mlsanity.com/paper/2307.11958/) (2023)
- [Exploring Model Transferability through the Lens of Potential Energy](https://mlsanity.com/paper/2308.15074/) (2023)
- [The Performance of Transferability Metrics does not Translate to Medical Tasks](https://mlsanity.com/paper/2308.07444/) (2023)
- [Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments](https://mlsanity.com/paper/2403.13803/) (2024)
- [Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components](https://mlsanity.com/paper/2412.19085/) (2024)
- [Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs](https://mlsanity.com/paper/2107.03815/) (2021)
- [On Characterizing the Evolution of Embedding Space of Neural Networks using Algebraic Topology](https://mlsanity.com/paper/2311.04592/) (2023)
- [Simple Transferability Estimation for Regression Tasks](https://mlsanity.com/paper/2312.00656/) (2023)
- [Sequence Transferability and Task Order Selection in Continual Learning](https://mlsanity.com/paper/2502.06544/) (2025)
- [Exploring Structural Degradation in Dense Representations for Self-supervised Learning](https://mlsanity.com/paper/2510.17299/) (2025)
- [How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey](https://mlsanity.com/paper/2312.04775/) (2023)
- [Exploiting a Zoo of Checkpoints for Unseen Tasks](https://mlsanity.com/paper/2111.03628/) (2021)
- [LEAD: Exploring Logit Space Evolution for Model Selection](https://mlsanity.com/paper/2507.14559/) (2025)
- [Transferability Metrics for Object Detection](https://mlsanity.com/paper/2306.15306/) (2023)
- [Feature Space Perturbation: A Panacea to Enhanced Transferability Estimation](https://mlsanity.com/paper/2502.16471/) (2025)
- [Benchmarking Transferability: A Framework for Fair and Robust Evaluation](https://mlsanity.com/paper/2504.20121/) (2025)

## Abstract (arXiv)

This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task. Thanks to the broad use of DNNs, we may easily collect hundreds of checkpoints from various sources. Which of them transfers the best to our downstream task of interest? Striving to answer this question thoroughly, we establish a neural checkpoint ranking benchmark (NeuCRaB) and study some intuitive ranking measures. These measures are generic, applying to the checkpoints of different output types without knowing how the checkpoints are pre-trained on which dataset. They also incur low computation cost, making them practically meaningful. Our results suggest that the linear separability of the features extracted by the checkpoints is a strong indicator of transferability. We also arrive at a new ranking measure, NLEEP, which gives rise to the best performance in the experiments.