# Deep Ensembles from a Bayesian Perspective

arXiv: https://arxiv.org/abs/2105.13283 · HTML: https://mlsanity.com/paper/2105.13283/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 51 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Explainability Techniques for Graph Convolutional Networks](https://mlsanity.com/paper/1905.13686/) (2019)
- [Hyperparameter Ensembles for Robustness and Uncertainty Quantification](https://mlsanity.com/paper/2006.13570/) (2020)
- [Uncertainty in Neural Networks: Approximately Bayesian Ensembling](https://mlsanity.com/paper/1810.05546/) (2018)
- [Bayesian Deep Ensembles via the Neural Tangent Kernel](https://mlsanity.com/paper/2007.05864/) (2020)
- [Evaluating and Calibrating Uncertainty Prediction in Regression Tasks](https://mlsanity.com/paper/1905.11659/) (2019)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [Quality of Uncertainty Quantification for Bayesian Neural Network Inference](https://mlsanity.com/paper/1906.09686/) (2019)
- [Testing Deep Neural Networks](https://mlsanity.com/paper/1803.04792/) (2018)
- [Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms](https://mlsanity.com/paper/2004.10710/) (2020)
- [Early Stopping is Nonparametric Variational Inference](https://mlsanity.com/paper/1504.01344/) (2015)
- [On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation](https://mlsanity.com/paper/2001.08049/) (2020)
- [The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning](https://mlsanity.com/paper/1811.12535/) (2018)
- [Uncertainty Quantification by Ensemble Learning for Computational Optical Form Measurements](https://mlsanity.com/paper/2103.01259/) (2021)

## Built on by (most cited first, 27 of 27)

- [Deep Ensembles Work, But Are They Necessary?](https://mlsanity.com/paper/2202.06985/) (2022)
- [Uncertainty quantification in fine-tuned LLMs using LoRA ensembles](https://mlsanity.com/paper/2402.12264/) (2024)
- [On Information-Theoretic Measures of Predictive Uncertainty](https://mlsanity.com/paper/2410.10786/) (2024)
- [Zero-Shot Uncertainty Quantification using Diffusion Probabilistic Models](https://mlsanity.com/paper/2408.04718/) (2024)
- [Finite-time Lyapunov exponents of deep neural networks](https://mlsanity.com/paper/2306.12548/) (2023)
- [Uncertainty Quantification with the Empirical Neural Tangent Kernel](https://mlsanity.com/paper/2502.02870/) (2025)
- [Graph Neural Network Interatomic Potential Ensembles with Calibrated Aleatoric and Epistemic Uncertainty on Energy and Forces](https://mlsanity.com/paper/2305.16325/) (2023)
- [A prediction rigidity formalism for low-cost uncertainties in trained neural networks](https://mlsanity.com/paper/2403.02251/) (2024)
- [Bayesian Neural Networks: Essentials](https://mlsanity.com/paper/2106.13594/) (2021)
- [Ensembling over Classifiers: a Bias-Variance Perspective](https://mlsanity.com/paper/2206.10566/) (2022)
- [Bayesian Deep Learning with Multilevel Trace-class Neural Networks](https://mlsanity.com/paper/2203.12961/) (2022)
- [Deep interpretable ensembles](https://mlsanity.com/paper/2205.12729/) (2022)
- [Bayesian posterior approximation with stochastic ensembles](https://mlsanity.com/paper/2212.08123/) (2022)
- [A framework for benchmarking uncertainty in deep regression](https://mlsanity.com/paper/2109.09048/) (2021)
- [Deep Gaussian Mixture Ensembles](https://mlsanity.com/paper/2306.07235/) (2023)
- [Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control](https://mlsanity.com/paper/2207.13730/) (2022)
- [DADEE: Well-calibrated uncertainty quantification in neural networks for barriers-based robot safety](https://mlsanity.com/paper/2407.00616/) (2024)
- [Contextual Similarity Distillation: Ensemble Uncertainties with a Single Model](https://mlsanity.com/paper/2503.11339/) (2025)
- [Looking at the posterior: accuracy and uncertainty of neural-network predictions](https://mlsanity.com/paper/2211.14605/) (2022)
- [Confident Naturalness Explanation (CNE): A Framework to Explain and Assess Patterns Forming Naturalness](https://mlsanity.com/paper/2311.08936/) (2023)
- [DBCal: Density Based Calibration of classifier predictions for uncertainty quantification](https://mlsanity.com/paper/2204.00150/) (2022)
- [A deep ensemble approach to X-ray polarimetry](https://mlsanity.com/paper/2111.03047/) (2021)
- [Coherent energy and force uncertainty in deep learning force fields](https://mlsanity.com/paper/2312.04174/) (2023)
- [A Priori Uncertainty Quantification of Reacting Turbulence Closure Models using Bayesian Neural Networks](https://mlsanity.com/paper/2402.18729/) (2024)
- [Evaluating structural uncertainty in accelerated MRI: are voxelwise measures useful surrogates?](https://mlsanity.com/paper/2503.10527/) (2025)
- [EMoE: Training-Free Expert Disagreement for Uncertainty-Aware Text-to-Image Diffusion](https://mlsanity.com/paper/2505.13273/) (2025)
- [Quantifying Out-of-Training Uncertainty of Neural-Network based Turbulence Closures](https://mlsanity.com/paper/2508.16891/) (2025)

## Abstract (arXiv)

Deep ensembles can be considered as the current state-of-the-art for uncertainty quantification in deep learning. While the approach was originally proposed as a non-Bayesian technique, arguments supporting its Bayesian footing have been put forward as well. We show that deep ensembles can be viewed as an approximate Bayesian method by specifying the corresponding assumptions. Our findings lead to an improved approximation which results in an enlarged epistemic part of the uncertainty. Numerical examples suggest that the improved approximation can lead to more reliable uncertainties. Analytical derivations ensure easy calculation of results.