mlsanityData to 14 Oct 2026

Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors

arXiv 2205.10279 ↗First posted 20 May 2022cs.LG cs.CVcited by 43

Not on the map. Why?

Abstract (arXiv)

Deep learning is increasingly moving towards a transfer learning paradigm whereby large foundation models are fine-tuned on downstream tasks, starting from an initialization learned on the source task. But an initialization contains relatively little information about the source task. Instead, we show that we can learn highly informative posteriors from the source task, through supervised or self-supervised approaches, which then serve as the basis for priors that modify the whole loss surface on the downstream task. This simple modular approach enables significant performance gains and more data-efficient learning on a variety of downstream classification and segmentation tasks, serving as a drop-in replacement for standard pre-training strategies. These highly informative priors also can be saved for future use, similar to pre-trained weights, and stand in contrast to the zero-mean isotropic uninformative priors that are typically used in Bayesian deep learning.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

35 of 50 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  11. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  12. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
23 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  2. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  3. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  4. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  5. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  6. 2017Variational Continual Learningcited by 339
  7. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  8. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  9. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  10. 2021Bayesian Neural Network Priors Revisitedcited by 121
  11. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
  12. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 102
  13. 2019Understanding Generalization through Visualizationscited by 54
  14. 2018Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variancecited by 34
  15. 2021Fast Adaptation with Linearized Neural Networkscited by 32
  16. 2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
  17. 2020Variational Auto-Regressive Gaussian Processes for Continual Learningcited by 23
  18. 2022Information Flow in Deep Neural Networkscited by 22
  19. 2018Optimal Bayesian Transfer Learningcited by 17
  20. 2020The Dual Information Bottleneckcited by 14
  21. 2020Semi-Supervised Learning with Variational Bayesian Inference and Maximum Uncertainty Regularizationcited by 3
  22. 2021Bayesian Transfer Learning: An Overview of Probabilistic Graphical Models for Transfer Learningcited by 2
  23. 2021Semi-supervised learning with Bayesian Confidence Propagation Neural Networkcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 16; 2024: 19; 2025: 4; 2026: 2.

43 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 250
  2. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  3. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  4. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  5. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 48
  6. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  7. 2023Reverse Engineering Self-Supervised Learningcited by 33
  8. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  9. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  10. 2023Variance-Covariance Regularization Improves Representation Learningcited by 14
  11. 2022Transfer learning for chemically accurate interatomic neural network potentialscited by 11
  12. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
31 more
  1. 2024Bayesian Neural Networks with Domain Knowledge Priorscited by 8
  2. 2023Uncertainty Quantification in Machine Learning for Biosignal Applications -- A Reviewcited by 7
  3. 2024Transferring Knowledge from Large Foundation Models to Small Downstream Modelscited by 7
  4. 2024Understanding the Role of Equivariance in Self-supervised Learningcited by 7
  5. 2023To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learningcited by 6
  6. 2024Learning from Noisy Labels via Conditional Distributionally Robust Optimizationcited by 6
  7. 2024Enhancing Transfer Learning with Flexible Nonparametric Posterior Samplingcited by 5
  8. 2024In-Context Symmetries: Self-Supervised Learning through Contextual World Modelscited by 5
  9. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4
  10. 2023Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networkscited by 4
  11. 2023Bayesian Transfer Learningcited by 3
  12. 2024Bayesian Multi-Task Transfer Learning for Soft Prompt Tuningcited by 3
  13. 2025Uncertainty-Aware Trajectory Prediction via Rule-Regularized Heteroscedastic Deep Classificationcited by 3
  14. 2024Informed Spectral Normalized Gaussian Processes for Trajectory Predictioncited by 2
  15. 2024Probabilistic Multi-Layer Perceptrons for Wind Farm Condition Monitoringcited by 2
  16. 2024Attacking Bayes: On the Adversarial Robustness of Bayesian Neural Networkscited by 2
  17. 2024Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reportedcited by 2
  18. 2025Learning Hyperparameters via a Data-Emphasized Variational Objectivecited by 2
  19. 2022Understanding Approximation for Bayesian Inference in Neural Networkscited by 1
  20. 2023Image Reconstruction via Deep Image Prior Subspacescited by 1
  21. 2024Domain Generalization Guided by Large-Scale Pre-Trained Priorscited by 1
  22. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  23. 2024Learning the Regularization Strength for Deep Fine-Tuning via a Data-Emphasized Variational Objectivecited by 1
  24. 2025Geometric Autoencoder Priors for Bayesian Inversion: Learn First Observe Latercited by 1
  25. 2023A Probabilistic Approach to Self-Supervised Learning using Cyclical Stochastic Gradient MCMCcited by 0
  26. 2024Mind the GAP: Improving Robustness to Subpopulation Shifts with Group-Aware Priorscited by 0
  27. 2024Multi-modal Speech Emotion Recognition via Feature Distribution Adaptation Networkcited by 0
  28. 2024Multivariate and Online Transfer Learning with Uncertainty Quantificationcited by 0
  29. 2025On-the-Fly Fine-Tuning of Foundational Neural Network Potentials: A Bayesian Neural Network Approachcited by 0
  30. 2026Amortising Inference and Meta-Learning Priors in Neural Networkscited by 0
  31. 2026Bridging Data Gaps in Structural Fragility Modeling through Transfer Learning: Methodology and Case Studiescited by 0