# Learning the Regularization Strength for Deep Fine-Tuning via a Data-Emphasized Variational Objective

arXiv: https://arxiv.org/abs/2410.19675 · HTML: https://mlsanity.com/paper/2410.19675/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It](https://mlsanity.com/paper/1412.3730/) (2014)
- [Bayesian Model Selection, the Marginal Likelihood, and Generalization](https://mlsanity.com/paper/2202.11678/) (2022)
- [Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization](https://mlsanity.com/paper/1805.10939/) (2018)
- [On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification](https://mlsanity.com/paper/2203.16481/) (2022)
- [Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors](https://mlsanity.com/paper/2205.10279/) (2022)
- [Variance reduction properties of the reparameterization trick](https://mlsanity.com/paper/1809.10330/) (2018)
- [Robust Bayesian inference via coarsening](https://mlsanity.com/paper/1506.06101/) (2015)
- [Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported](https://mlsanity.com/paper/2405.15583/) (2024)

## Built on by (most cited first, 1 of 1)

- [Occam's Razor is Only as Sharp as Your ELBO](https://mlsanity.com/paper/2604.25984/) (2026)

## Abstract (arXiv)

A number of popular transfer learning methods rely on grid search to select regularization hyperparameters that control over-fitting. This grid search requirement has several key disadvantages: the search is computationally expensive, requires carving out a validation set that reduces the size of available data for model training, and requires practitioners to specify candidate values. In this paper, we propose an alternative to grid search: directly learning regularization hyperparameters on the full training set via model selection techniques based on the evidence lower bound ("ELBo") objective from variational methods. For deep neural networks with millions of parameters, we specifically recommend a modified ELBo that upweights the influence of the data likelihood relative to the prior while remaining a valid bound on the evidence for Bayesian model selection. Our proposed technique overcomes all three disadvantages of grid search. We demonstrate effectiveness on image classification tasks on several datasets, yielding heldout accuracy comparable to existing approaches with far less compute time.