# A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods

arXiv: https://arxiv.org/abs/2305.15027 · HTML: https://mlsanity.com/paper/2305.15027/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 77 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 30 of 37)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [A General Framework for Updating Belief Distributions](https://mlsanity.com/paper/1306.6430/) (2013)
- [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017)
- [Stein Variational Gradient Descent as Gradient Flow](https://mlsanity.com/paper/1704.07520/) (2017)
- [A Primer on PAC-Bayesian Learning](https://mlsanity.com/paper/1901.05353/) (2019)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Maximum Mean Discrepancy Gradient Flow](https://mlsanity.com/paper/1906.04370/) (2019)
- [Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks](https://mlsanity.com/paper/1701.04722/) (2017)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [The Case for Bayesian Deep Learning](https://mlsanity.com/paper/2001.10995/) (2020)
- [Gradient Estimators for Implicit Models](https://mlsanity.com/paper/1705.07107/) (2017)
- [Kernel Stein Discrepancy Descent](https://mlsanity.com/paper/2105.09994/) (2021)
- [Robust Generalised Bayesian Inference for Intractable Likelihoods](https://mlsanity.com/paper/2104.07359/) (2021)
- [Simpler PAC-Bayesian Bounds for Hostile Data](https://mlsanity.com/paper/1610.07193/) (2016)
- [A Spectral Approach to Gradient Estimation for Implicit Distributions](https://mlsanity.com/paper/1806.02925/) (2018)
- [Variational Implicit Processes](https://mlsanity.com/paper/1806.02390/) (2018)
- [KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support](https://mlsanity.com/paper/2106.08929/) (2021)
- [Scaling limit of the Stein variational gradient descent: the mean field regime](https://mlsanity.com/paper/1805.04035/) (2018)
- [Proximal Langevin Algorithm: Rapid Convergence Under Isoperimetry](https://mlsanity.com/paper/1911.01469/) (2019)
- [Robust Bayesian inference via coarsening](https://mlsanity.com/paper/1506.06101/) (2015)

## Built on by (most cited first, 30 of 32)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcuts](https://mlsanity.com/paper/2402.12240/) (2024)
- [Uncertainty quantification in fine-tuned LLMs using LoRA ensembles](https://mlsanity.com/paper/2402.12264/) (2024)
- [Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approach](https://mlsanity.com/paper/2506.03979/) (2025)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning](https://mlsanity.com/paper/2402.04005/) (2024)
- [TabMGP: Martingale Posterior with TabPFN](https://mlsanity.com/paper/2510.25154/) (2025)
- [Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles](https://mlsanity.com/paper/2412.15758/) (2024)
- [Ensembled Prediction Intervals for Causal Outcomes Under Hidden Confounding](https://mlsanity.com/paper/2306.09520/) (2023)
- [Are Ensembles Getting Better all the Time?](https://mlsanity.com/paper/2311.17885/) (2023)
- [A Quadrature Approach for General-Purpose Batch Bayesian Optimization via Probabilistic Lifting](https://mlsanity.com/paper/2404.12219/) (2024)
- [Rethinking the Diffusion Models for Numerical Tabular Data Imputation from the Perspective of Wasserstein Gradient Flow](https://mlsanity.com/paper/2406.15762/) (2024)
- [Multi-Output Robust and Conjugate Gaussian Processes](https://mlsanity.com/paper/2510.26401/) (2025)
- [On the Epistemic Uncertainty of Overparametrized Neural Networks](https://mlsanity.com/paper/2605.25234/) (2026)
- [Trust Me, I Know the Way: Predictive Uncertainty in the Presence of Shortcut Learning](https://mlsanity.com/paper/2502.09137/) (2025)
- [Robust Bayesian Optimisation with Unbounded Corruptions](https://mlsanity.com/paper/2511.15315/) (2025)
- [Outlier-robust Diffusion Posterior Sampling for Bayesian Inverse Problems](https://mlsanity.com/paper/2602.02045/) (2026)
- [Regret Analysis of Guided Diffusion for Black-Box Optimization over Structured Inputs](https://mlsanity.com/paper/2605.10385/) (2026)
- [Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning](https://mlsanity.com/paper/2605.21765/) (2026)
- [Thinned Mean Field Langevin Dynamics](https://mlsanity.com/paper/2605.28589/) (2026)
- [Bayesian Low-Rank LeArning (Bella): A Practical Approach to Bayesian Neural Networks](https://mlsanity.com/paper/2407.20891/) (2024)
- [Stein Variational Newton Neural Network Ensembles](https://mlsanity.com/paper/2411.01887/) (2024)
- [Evaluating structural uncertainty in accelerated MRI: are voxelwise measures useful surrogates?](https://mlsanity.com/paper/2503.10527/) (2025)
- [Do you understand epistemic uncertainty? Think again! Rigorous frequentist epistemic uncertainty estimation in regression](https://mlsanity.com/paper/2503.13317/) (2025)
- [Variational Deep Learning via Implicit Regularization](https://mlsanity.com/paper/2505.20235/) (2025)
- [PEdger++: Practical Edge Detection via Assembling Cross Information](https://mlsanity.com/paper/2508.11961/) (2025)
- [Rates of Convergence of Generalised Variational Inference Posteriors under Prior Misspecification](https://mlsanity.com/paper/2510.03109/) (2025)
- [Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformers](https://mlsanity.com/paper/2605.03614/) (2026)
- [Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Training](https://mlsanity.com/paper/2605.27747/) (2026)
- [Uncertainty quantification for trustworthy deep learning: Methods and measures](https://mlsanity.com/paper/2607.28248/) (2026)

## Abstract (arXiv)

We establish the first mathematically rigorous link between Bayesian, variational Bayesian, and ensemble methods. A key step towards this it to reformulate the non-convex optimisation problem typically encountered in deep learning as a convex optimisation in the space of probability measures. On a technical level, our contribution amounts to studying generalised variational inference through the lense of Wasserstein gradient flows. The result is a unified theory of various seemingly disconnected approaches that are commonly used for uncertainty quantification in deep learning -- including deep ensembles and (variational) Bayesian methods. This offers a fresh perspective on the reasons behind the success of deep ensembles over procedures based on parameterised variational inference, and allows the derivation of new ensembling schemes with convergence guarantees. We showcase this by proposing a family of interacting deep ensembles with direct parallels to the interactions of particle systems in thermodynamics, and use our theory to prove the convergence of these algorithms to a well-defined global minimiser on the space of probability measures.