# Bayesian Deep Ensembles via the Neural Tangent Kernel

arXiv: https://arxiv.org/abs/2007.05864 · HTML: https://mlsanity.com/paper/2007.05864/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2020.

## Built on by (most cited first, 30 of 90)

- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation](https://mlsanity.com/paper/2110.03051/) (2021)
- [Deep Ensembles Work, But Are They Necessary?](https://mlsanity.com/paper/2202.06985/) (2022)
- [Fast Finite Width Neural Tangent Kernel](https://mlsanity.com/paper/2206.08720/) (2022)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs](https://mlsanity.com/paper/2106.04110/) (2021)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [PINNACLE: PINN Adaptive ColLocation and Experimental points selection](https://mlsanity.com/paper/2404.07662/) (2024)
- [A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel](https://mlsanity.com/paper/2206.12543/) (2022)
- [Deep Ensembles from a Bayesian Perspective](https://mlsanity.com/paper/2105.13283/) (2021)
- [CreINNs: Credal-Set Interval Neural Networks for Uncertainty Estimation in Classification Tasks](https://mlsanity.com/paper/2401.05043/) (2024)
- [Sample-Then-Optimize Batch Neural Thompson Sampling](https://mlsanity.com/paper/2210.06850/) (2022)
- [Contrasting random and learned features in deep Bayesian linear regression](https://mlsanity.com/paper/2203.00573/) (2022)
- [The Neural Testbed: Evaluating Joint Predictions](https://mlsanity.com/paper/2110.04629/) (2021)
- [Wide Mean-Field Bayesian Neural Networks Ignore the Data](https://mlsanity.com/paper/2202.11670/) (2022)
- [On the infinite-depth limit of finite-width neural networks](https://mlsanity.com/paper/2210.00688/) (2022)
- [Bayesian Interpolation with Deep Linear Networks](https://mlsanity.com/paper/2212.14457/) (2022)
- [Width and Depth Limits Commute in Residual Networks](https://mlsanity.com/paper/2302.00453/) (2023)
- [Single Model Uncertainty Estimation via Stochastic Data Centering](https://mlsanity.com/paper/2207.07235/) (2022)
- [Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2405.03425/) (2024)
- [UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomography](https://mlsanity.com/paper/2202.10847/) (2022)
- [Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping](https://mlsanity.com/paper/2206.03633/) (2022)
- [Improving Neural Additive Models with Bayesian Principles](https://mlsanity.com/paper/2305.16905/) (2023)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Conditional Variational Autoencoder for Learned Image Reconstruction](https://mlsanity.com/paper/2110.11681/) (2021)
- [Credal Wrapper of Model Averaging for Uncertainty Estimation in Classification](https://mlsanity.com/paper/2405.15047/) (2024)

## Abstract (arXiv)

We explore the link between deep ensembles and Gaussian processes (GPs) through the lens of the Neural Tangent Kernel (NTK): a recent development in understanding the training dynamics of wide neural networks (NNs). Previous work has shown that even in the infinite width limit, when NNs become GPs, there is no GP posterior interpretation to a deep ensemble trained with squared error loss. We introduce a simple modification to standard deep ensembles training, through addition of a computationally-tractable, randomised and untrainable function to each ensemble member, that enables a posterior interpretation in the infinite width limit. When ensembled together, our trained NNs give an approximation to a posterior predictive distribution, and we prove that our Bayesian deep ensembles make more conservative predictions than standard deep ensembles in the infinite width limit. Finally, using finite width NNs we demonstrate that our Bayesian deep ensembles faithfully emulate the analytic posterior predictive when available, and can outperform standard deep ensembles in various out-of-distribution settings, for both regression and classification tasks.