mlsanityData to 7 Oct 2026

A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods

arXiv 2305.15027 ↗First posted 24 May 2023stat.ML cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

We establish the first mathematically rigorous link between Bayesian, variational Bayesian, and ensemble methods. A key step towards this it to reformulate the non-convex optimisation problem typically encountered in deep learning as a convex optimisation in the space of probability measures. On a technical level, our contribution amounts to studying generalised variational inference through the lense of Wasserstein gradient flows. The result is a unified theory of various seemingly disconnected approaches that are commonly used for uncertainty quantification in deep learning -- including deep ensembles and (variational) Bayesian methods. This offers a fresh perspective on the reasons behind the success of deep ensembles over procedures based on parameterised variational inference, and allows the derivation of new ensembling schemes with convergence guarantees. We showcase this by proposing a family of interacting deep ensembles with direct parallels to the interactions of particle systems in thermodynamics, and use our theory to prove the convergence of these algorithms to a well-defined global minimiser on the space of probability measures.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 37 of its 77 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

37 of 77 references.

References · 37

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2015Variational Inference with Normalizing Flowscited by 2,161
  4. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  5. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  6. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  7. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  8. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 360
  9. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  10. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  11. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  12. 2013A General Framework for Updating Belief Distributionscited by 172
25 more
  1. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  2. 2017Stein Variational Gradient Descent as Gradient Flowcited by 149
  3. 2019A Primer on PAC-Bayesian Learningcited by 130
  4. 2019Functional Variational Bayesian Neural Networkscited by 127
  5. 2019Maximum Mean Discrepancy Gradient Flowcited by 123
  6. 2017Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networkscited by 107
  7. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  8. 2020The Case for Bayesian Deep Learningcited by 68
  9. 2017Gradient Estimators for Implicit Modelscited by 58
  10. 2021Kernel Stein Discrepancy Descentcited by 57
  11. 2021Robust Generalised Bayesian Inference for Intractable Likelihoodscited by 53
  12. 2016Simpler PAC-Bayesian Bounds for Hostile Datacited by 45
  13. 2018A Spectral Approach to Gradient Estimation for Implicit Distributionscited by 45
  14. 2018Variational Implicit Processescited by 41
  15. 2021KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Supportcited by 37
  16. 2018Scaling limit of the Stein variational gradient descent: the mean field regimecited by 35
  17. 2015Robust Bayesian inference via coarseningcited by 32
  18. 2019Proximal Langevin Algorithm: Rapid Convergence Under Isoperimetrycited by 31
  19. 2020A comparison of learning rate selection methods in generalized Bayesian inferencecited by 23
  20. 2018Doubly Robust Bayesian Inference for Non-Stationary Streaming Data with β-Divergencescited by 21
  21. 2022Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learningcited by 20
  22. 2020Non-exponentially weighted aggregation: regret bounds for unbounded loss functionscited by 18
  23. 2023Robust and Scalable Bayesian Online Changepoint Detectioncited by 16
  24. 2021Function-space Inference with Sparse Implicit Processescited by 13
  25. 2019Frequentist Consistency of Generalized Variational Inferencecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 11; 2026: 9.

31 papers.

Sort:

By area:

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2024BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcutscited by 36
  3. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
  4. 2025Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approachcited by 18
  5. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  6. 2025Uncertainty Quantification for Machine Learning in Healthcare: A Surveycited by 11
  7. 2024Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensemblescited by 8
  8. 2023Ensembled Prediction Intervals for Causal Outcomes Under Hidden Confoundingcited by 7
  9. 2023Are Ensembles Getting Better all the Time?cited by 6
  10. 2025TabMGP: Martingale Posterior with TabPFNcited by 6
  11. 2024A Quadrature Approach for General-Purpose Batch Bayesian Optimization via Probabilistic Liftingcited by 3
  12. 2024Rethinking the Diffusion Models for Numerical Tabular Data Imputation from the Perspective of Wasserstein Gradient Flowcited by 3
19 more
  1. 2026On the Epistemic Uncertainty of Overparametrized Neural Networkscited by 2
  2. 2025Trust Me, I Know the Way: Predictive Uncertainty in the Presence of Shortcut Learningcited by 1
  3. 2025Multi-Output Robust and Conjugate Gaussian Processescited by 1
  4. 2025Robust Bayesian Optimisation with Unbounded Corruptionscited by 1
  5. 2026Outlier-robust Diffusion Posterior Sampling for Bayesian Inverse Problemscited by 1
  6. 2026Regret Analysis of Guided Diffusion for Black-Box Optimization over Structured Inputscited by 1
  7. 2026Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learningcited by 1
  8. 2026Thinned Mean Field Langevin Dynamicscited by 1
  9. 2024Bayesian Low-Rank LeArning (Bella): A Practical Approach to Bayesian Neural Networkscited by 0
  10. 2024Stein Variational Newton Neural Network Ensemblescited by 0
  11. 2025Evaluating structural uncertainty in accelerated MRI: are voxelwise measures useful surrogates?cited by 0
  12. 2025Do you understand epistemic uncertainty? Think again! Rigorous frequentist epistemic uncertainty estimation in regressioncited by 0
  13. 2025Variational Deep Learning via Implicit Regularizationcited by 0
  14. 2025PEdger++: Practical Edge Detection via Assembling Cross Informationcited by 0
  15. 2025Rates of Convergence of Generalised Variational Inference Posteriors under Prior Misspecificationcited by 0
  16. 2026Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformerscited by 0
  17. 2026Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Trainingcited by 0
  18. 2026Uncertainty quantification for trustworthy deep learning: Methods and measurescited by 0
  19. 2026Continuous Variational Synthesiscited by 0