# Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

arXiv: https://arxiv.org/abs/2006.08573 · HTML: https://mlsanity.com/paper/2006.08573/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 30 of 56)

- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking](https://mlsanity.com/paper/2312.09244/) (2023)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields](https://mlsanity.com/paper/2309.03185/) (2023)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [Deep Ensembles Work, But Are They Necessary?](https://mlsanity.com/paper/2202.06985/) (2022)
- [No One Representation to Rule Them All: Overlapping Features of Training Methods](https://mlsanity.com/paper/2110.12899/) (2021)
- [Uncertainty Estimation by Fisher Information-based Evidential Deep Learning](https://mlsanity.com/paper/2303.02045/) (2023)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification](https://mlsanity.com/paper/2110.13511/) (2021)
- [Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks](https://mlsanity.com/paper/2412.15238/) (2024)
- [UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomography](https://mlsanity.com/paper/2202.10847/) (2022)
- [Revisiting the Importance of Amplifying Bias for Debiasing](https://mlsanity.com/paper/2205.14594/) (2022)
- [Calibrating Multimodal Learning](https://mlsanity.com/paper/2306.01265/) (2023)
- [How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?](https://mlsanity.com/paper/2302.03679/) (2023)
- [FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling](https://mlsanity.com/paper/2303.00586/) (2023)
- [Deep-Ensemble-Based Uncertainty Quantification in Spatiotemporal Graph Neural Networks for Traffic Forecasting](https://mlsanity.com/paper/2204.01618/) (2022)
- [Out-of-Distribution Detection via Deep Multi-Comprehension Ensemble](https://mlsanity.com/paper/2403.16260/) (2024)
- [Neural Ensemble Search via Bayesian Sampling](https://mlsanity.com/paper/2109.02533/) (2021)
- [Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation](https://mlsanity.com/paper/2206.15047/) (2022)
- [DivBO: Diversity-aware CASH for Ensemble Learning](https://mlsanity.com/paper/2302.03255/) (2023)
- [Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification](https://mlsanity.com/paper/2304.04042/) (2023)
- [Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs](https://mlsanity.com/paper/2110.10645/) (2021)
- [Bayesian posterior approximation with stochastic ensembles](https://mlsanity.com/paper/2212.08123/) (2022)
- [EPiC: Ensemble of Partial Point Clouds for Robust Classification](https://mlsanity.com/paper/2303.11419/) (2023)
- [Towards Less Constrained Macro-Neural Architecture Search](https://mlsanity.com/paper/2203.05508/) (2022)
- [PSEO: Optimizing Post-hoc Stacking Ensemble Through Hyperparameter Tuning](https://mlsanity.com/paper/2508.05144/) (2025)

## Abstract (arXiv)

Ensembles of neural networks achieve superior performance compared to stand-alone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift. Deep ensembles, a state-of-the-art method for uncertainty estimation, only ensemble random initializations of a fixed architecture. Instead, we propose two methods for automatically constructing ensembles with varying architectures, which implicitly trade-off individual architectures' strengths against the ensemble's diversity and exploit architectural variation as a source of diversity. On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift. Our further analysis and ablation studies provide evidence of higher ensemble diversity due to architectural variation, resulting in ensembles that can outperform deep ensembles, even when having weaker average base learners. To foster reproducibility, our code is available: urlhttps://github.com/automl/nes