mlsanityData to 14 Oct 2026

A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints

arXiv 2410.05612 ↗First posted 8 Oct 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent advances in artificial intelligence have been fueled by the development of foundation models such as BERT, GPT, T5, and Vision Transformers. These models are first pretrained on vast and diverse datasets and then adapted to specific downstream tasks, often with significantly less data. However, the mechanisms behind the success of this ubiquitous pretrain-then-adapt paradigm remain underexplored, particularly the characteristics of pretraining checkpoints that enhance downstream adaptation. We introduce a Bayesian model selection criterion, called the downstream free energy, which quantifies a checkpoint's adaptability by measuring the concentration of nearby favorable parameters for the downstream task. We demonstrate that this Bayesian model selection criterion can be effectively implemented without access to the downstream data or prior knowledge of the downstream task. Furthermore, we provide empirical evidence that the criterion reliably correlates with improved finetuning performance, offering a principled approach to predicting model adaptability.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

29 of 44 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  9. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  10. 2018Do Better ImageNet Models Transfer Better?cited by 565
  11. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  12. 2018Meta-learning with differentiable closed-form solverscited by 397
17 more
  1. 2019A Baseline for Few-Shot Image Classificationcited by 298
  2. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  3. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  4. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  5. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  6. 2021A Survey of Visual Transformerscited by 117
  7. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
  8. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  9. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  10. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  11. 2021CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretationcited by 57
  12. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  13. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  14. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  15. 2023Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challengescited by 18
  16. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  17. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Stagewise Reinforcement Learning and the Geometry of the Regret Landscapecited by 4
  2. 2026Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorizationcited by 0
  3. 2026Target-Independent Micro-Interventions for Predicting Training Response Across Language-Model Familiescited by 0
  4. 2026From Checkpoint Variation to Selection Gains in Supervised Fine-Tuningcited by 0
  5. 2026Why Heavy-Tailed Weight Spectra Predict Model Qualitycited by 0