# A Bayesian Perspective on Training Speed and Model Selection

arXiv: https://arxiv.org/abs/2010.14499 · HTML: https://mlsanity.com/paper/2010.14499/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning](https://mlsanity.com/paper/2104.04975/) (2021)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Bayesian Model Selection, the Marginal Likelihood, and Generalization](https://mlsanity.com/paper/2202.11678/) (2022)
- [Deep Active Learning by Leveraging Training Dynamics](https://mlsanity.com/paper/2110.08611/) (2021)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Early Stopping Against Label Noise Without Validation Data](https://mlsanity.com/paper/2502.07551/) (2025)
- [Hyperparameter Optimization through Neural Network Partitioning](https://mlsanity.com/paper/2304.14766/) (2023)
- [Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions](https://mlsanity.com/paper/2401.04305/) (2024)
- [A User-Guided Bayesian Framework for Ensemble Feature Selection in Life Science Applications (UBayFS)](https://mlsanity.com/paper/2104.14787/) (2021)
- [Marginal and Joint Cross-Entropies & Predictives for Online Bayesian Inference, Active Learning, and Active Sampling](https://mlsanity.com/paper/2205.08766/) (2022)
- [Cold Posteriors through PAC-Bayes](https://mlsanity.com/paper/2206.11173/) (2022)
- [Leveraging Unlabeled Data to Track Memorization](https://mlsanity.com/paper/2212.04461/) (2022)
- [Fine-Tuned In-Context Learners for Efficient Adaptation](https://mlsanity.com/paper/2512.19879/) (2025)
- [Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection](https://mlsanity.com/paper/2107.00191/) (2021)
- [SAVA: Scalable Learning-Agnostic Data Valuation](https://mlsanity.com/paper/2406.01130/) (2024)
- [DARTS without a Validation Set: Optimizing the Marginal Likelihood](https://mlsanity.com/paper/2112.13023/) (2021)
- [Is Supervised Learning Really That Different from Unsupervised?](https://mlsanity.com/paper/2505.11006/) (2025)
- [Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlo](https://mlsanity.com/paper/2606.00293/) (2026)

## Abstract (arXiv)

We take a Bayesian perspective to illustrate a connection between training speed and the marginal likelihood in linear models. This provides two major insights: first, that a measure of a model's training speed can be used to estimate its marginal likelihood. Second, that this measure, under certain conditions, predicts the relative weighting of models in linear model combinations trained to minimize a regression loss. We verify our results in model selection tasks for linear models and for the infinite-width limit of deep neural networks. We further provide encouraging empirical evidence that the intuition developed in these settings also holds for deep neural networks trained with stochastic gradient descent. Our results suggest a promising new direction towards explaining why neural networks trained with stochastic gradient descent are biased towards functions that generalize well.