# Layer Ensembles

arXiv: https://arxiv.org/abs/2210.04882 · HTML: https://mlsanity.com/paper/2210.04882/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition](https://mlsanity.com/paper/1509.08971/) (2015)
- [Hypermodels for Exploration](https://mlsanity.com/paper/2006.07464/) (2020)
- [Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classification](https://mlsanity.com/paper/1910.08168/) (2019)
- [Layer Ensembles: A Single-Pass Uncertainty Estimation in Deep Learning for Segmentation](https://mlsanity.com/paper/2203.08878/) (2022)
- [Variational Neural Networks](https://mlsanity.com/paper/2207.01524/) (2022)

## Built on by (most cited first, 3 of 3)

- [Bayesian Learning for Neural Networks: an algorithmic survey](https://mlsanity.com/paper/2211.11865/) (2022)
- [Variational Graph Convolutional Neural Networks](https://mlsanity.com/paper/2507.01699/) (2025)
- [Uncertainty quantification for trustworthy deep learning: Methods and measures](https://mlsanity.com/paper/2607.28248/) (2026)

## Abstract (arXiv)

Deep Ensembles, as a type of Bayesian Neural Networks, can be used to estimate uncertainty on the prediction of multiple neural networks by collecting votes from each network and computing the difference in those predictions. In this paper, we introduce a method for uncertainty estimation that considers a set of independent categorical distributions for each layer of the network, giving many more possible samples with overlapped layers than in the regular Deep Ensembles. We further introduce an optimized inference procedure that reuses common layer outputs, achieving up to 19x speed up and reducing memory usage quadratically. We also show that the method can be further improved by ranking samples, resulting in models that require less memory and time to run while achieving higher uncertainty quality than Deep Ensembles.