mlsanityData to 7 Oct 2026

Data augmentation in Bayesian neural networks and the cold posterior effect

arXiv 2106.05586 ↗First posted 10 Jun 2021stat.ML cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Bayesian neural networks that incorporate data augmentation implicitly use a ``randomly perturbed log-likelihood [which] does not have a clean interpretation as a valid likelihood function'' (Izmailov et al. 2021). Here, we provide several approaches to developing principled Bayesian neural networks incorporating data augmentation. We introduce a ``finite orbit'' setting which allows likelihoods to be computed exactly, and give tight multi-sample bounds in the more usual ``full orbit'' setting. These models cast light on the origin of the cold posterior effect. In particular, we find that the cold posterior effect persists even in these principled models incorporating data augmentation. This suggests that the cold posterior effect cannot be dismissed as an artifact of data augmentation using incorrect likelihoods.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 36 of its 66 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

36 of 66 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2021Convex optimizationcited by 2,479
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2021Going deeper with Image Transformerscited by 759
  9. 2015Importance Weighted Autoencoderscited by 454
  10. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  11. 2019Human uncertainty makes classification more robustcited by 231
  12. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
24 more
  1. 2019Practical Deep Learning with Bayesian Principlescited by 158
  2. 2021Bayesian Neural Network Priors Revisitedcited by 121
  3. 2018A Kernel Theory of Modern Data Augmentationcited by 106
  4. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  5. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  6. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  7. 2014Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing Itcited by 92
  8. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  9. 2020On the Benefits of Invariance in Neural Networkscited by 78
  10. 2020A statistical theory of cold posteriors in deep neural networkscited by 53
  11. 2020Learning Invariances in Neural Networkscited by 52
  12. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  13. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  14. 2019Augment your batch: better training with larger batchescited by 37
  15. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32
  16. 2021Exact Langevin Dynamics with Stochastic Gradientscited by 30
  17. 2020Cold Posteriors and Aleatoric Uncertaintycited by 22
  18. 2021BNNpriors: A library for Bayesian neural network inference with different prior distributionscited by 20
  19. 2019Faster Neural Network Training with Data Echoingcited by 19
  20. 2020Improving Transformation Invariance in Contrastive Representation Learningcited by 16
  21. 2021Bayesian OOD detection with aleatoric uncertainty and outlier exposurecited by 16
  22. 2019Minimizers of the Empirical Risk and Risk Monotonicitycited by 11
  23. 2018Tensor Monte Carlo: particle methods for the GPU eracited by 8
  24. 2013Invariances of random fields paths, with applications in Gaussian Process Regressioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 8; 2024: 10; 2025: 4; 2026: 2.

36 papers.

Sort:

By area:

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  3. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  4. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  5. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  6. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  7. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  8. 2022UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographycited by 19
  9. 2023Improving Neural Additive Models with Bayesian Principlescited by 15
  10. 2022How Tempering Fixes Data Augmentation in Bayesian Neural Networkscited by 13
  11. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  12. 2022Regularising for invariance to data augmentation improves supervised learningcited by 9
24 more
  1. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  2. 2023Hyperparameter Optimization through Neural Network Partitioningcited by 8
  3. 2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
  4. 2021Bayesian neural network unit priors and generalized Weibull-tail propertycited by 5
  5. 2024Can a Confident Prior Replace a Cold Posterior?cited by 5
  6. 2022Learning Continuous Rotation Canonicalization with Radial Beam Samplingcited by 4
  7. 2022Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical studycited by 4
  8. 2024Noether's razor: Learning Conserved Quantitiescited by 4
  9. 2024Minimax optimality of deep neural networks on dependent data via PAC-Bayes boundscited by 4
  10. 2022Cold Posteriors through PAC-Bayescited by 3
  11. 2023The fine print on tempered posteriorscited by 3
  12. 2024On Uncertainty Quantification for Near-Bayes Optimal Algorithmscited by 3
  13. 2026Position: agentic AI orchestration should be Bayes-consistentcited by 3
  14. 2023If there is no underfitting, there is no Cold Posterior Effectcited by 2
  15. 2024Robust Classification by Coupling Data Mollification with Label Smoothingcited by 2
  16. 2025Parameter Expanded Stochastic Gradient Markov Chain Monte Carlocited by 2
  17. 2024Bayesian Uncertainty Estimation by Hamiltonian Monte Carlo: Applications to Cardiac MRI Segmentationcited by 1
  18. 2024Temperature Optimization for Bayesian Deep Learningcited by 1
  19. 2025Optimizing Data Augmentation through Bayesian Model Selectioncited by 1
  20. 2025Tempering the Bayes Filter towards Improved Model-Based Estimationcited by 1
  21. 2021Weight Pruning and Uncertainty in Radio Galaxy Classificationcited by 0
  22. 2023Uncertainty in Graph Contrastive Learning with Bayesian Neural Networkscited by 0
  23. 2024On Uniform, Bayesian, and PAC-Bayesian Deep Ensemblescited by 0
  24. 2026Gaussian Mean Field Variational Inference can Overestimate Predictive Variancecited by 0