# Data augmentation in Bayesian neural networks and the cold posterior effect

arXiv: https://arxiv.org/abs/2106.05586 · HTML: https://mlsanity.com/paper/2106.05586/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 66 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [A Kernel Theory of Modern Data Augmentation](https://mlsanity.com/paper/1803.06084/) (2018)
- [Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning](https://mlsanity.com/paper/2104.04975/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It](https://mlsanity.com/paper/1412.3730/) (2014)
- [Noisy Natural Gradient as Variational Inference](https://mlsanity.com/paper/1712.02390/) (2017)
- [MultiGrain: a unified image embedding for classes and instances](https://mlsanity.com/paper/1902.05509/) (2019)
- [On the Benefits of Invariance in Neural Networks](https://mlsanity.com/paper/2005.00178/) (2020)
- [A statistical theory of cold posteriors in deep neural networks](https://mlsanity.com/paper/2008.05912/) (2020)
- [Learning Invariances in Neural Networks](https://mlsanity.com/paper/2010.11882/) (2020)
- [Learning Invariances using the Marginal Likelihood](https://mlsanity.com/paper/1808.05563/) (2018)
- [Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes](https://mlsanity.com/paper/2005.08140/) (2020)
- [Augment your batch: better training with larger batches](https://mlsanity.com/paper/1901.09335/) (2019)
- [Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error](https://mlsanity.com/paper/2105.13343/) (2021)
- [Exact Langevin Dynamics with Stochastic Gradients](https://mlsanity.com/paper/2102.01691/) (2021)
- [Cold Posteriors and Aleatoric Uncertainty](https://mlsanity.com/paper/2008.00029/) (2020)
- [Faster Neural Network Training with Data Echoing](https://mlsanity.com/paper/1907.05550/) (2019)

## Built on by (most cited first, 30 of 36)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification](https://mlsanity.com/paper/2203.16481/) (2022)
- [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://mlsanity.com/paper/2202.10638/) (2022)
- [Adapting the Linearised Laplace Model Evidence for Modern Deep Learning](https://mlsanity.com/paper/2206.08900/) (2022)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Last Layer Marginal Likelihood for Invariance Learning](https://mlsanity.com/paper/2106.07512/) (2021)
- [Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2405.03425/) (2024)
- [UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomography](https://mlsanity.com/paper/2202.10847/) (2022)
- [Improving Neural Additive Models with Bayesian Principles](https://mlsanity.com/paper/2305.16905/) (2023)
- [How Tempering Fixes Data Augmentation in Bayesian Neural Networks](https://mlsanity.com/paper/2205.13900/) (2022)
- [Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization](https://mlsanity.com/paper/2304.08309/) (2023)
- [Regularising for invariance to data augmentation improves supervised learning](https://mlsanity.com/paper/2203.03304/) (2022)
- [Incorporating Unlabelled Data into Bayesian Neural Networks](https://mlsanity.com/paper/2304.01762/) (2023)
- [Hyperparameter Optimization through Neural Network Partitioning](https://mlsanity.com/paper/2304.14766/) (2023)
- [Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems](https://mlsanity.com/paper/2506.10060/) (2025)
- [Bayesian neural network unit priors and generalized Weibull-tail property](https://mlsanity.com/paper/2110.02885/) (2021)
- [Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study](https://mlsanity.com/paper/2207.03859/) (2022)
- [Can a Confident Prior Replace a Cold Posterior?](https://mlsanity.com/paper/2403.01272/) (2024)
- [Learning Continuous Rotation Canonicalization with Radial Beam Sampling](https://mlsanity.com/paper/2206.10690/) (2022)
- [Noether's razor: Learning Conserved Quantities](https://mlsanity.com/paper/2410.08087/) (2024)
- [Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds](https://mlsanity.com/paper/2410.21702/) (2024)
- [Cold Posteriors through PAC-Bayes](https://mlsanity.com/paper/2206.11173/) (2022)
- [The fine print on tempered posteriors](https://mlsanity.com/paper/2309.05292/) (2023)
- [On Uncertainty Quantification for Near-Bayes Optimal Algorithms](https://mlsanity.com/paper/2403.19381/) (2024)
- [Position: agentic AI orchestration should be Bayes-consistent](https://mlsanity.com/paper/2605.00742/) (2026)
- [If there is no underfitting, there is no Cold Posterior Effect](https://mlsanity.com/paper/2310.01189/) (2023)
- [Robust Classification by Coupling Data Mollification with Label Smoothing](https://mlsanity.com/paper/2406.01494/) (2024)
- [Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo](https://mlsanity.com/paper/2503.00699/) (2025)
- [Bayesian Uncertainty Estimation by Hamiltonian Monte Carlo: Applications to Cardiac MRI Segmentation](https://mlsanity.com/paper/2403.02311/) (2024)
- [Temperature Optimization for Bayesian Deep Learning](https://mlsanity.com/paper/2410.05757/) (2024)

## Abstract (arXiv)

Bayesian neural networks that incorporate data augmentation implicitly use a ``randomly perturbed log-likelihood [which] does not have a clean interpretation as a valid likelihood function'' (Izmailov et al. 2021). Here, we provide several approaches to developing principled Bayesian neural networks incorporating data augmentation. We introduce a ``finite orbit'' setting which allows likelihoods to be computed exactly, and give tight multi-sample bounds in the more usual ``full orbit'' setting. These models cast light on the origin of the cold posterior effect. In particular, we find that the cold posterior effect persists even in these principled models incorporating data augmentation. This suggests that the cold posterior effect cannot be dismissed as an artifact of data augmentation using incorrect likelihoods.