# Scalable Ensemble Diversification for OOD Generalization and Detection

arXiv: https://arxiv.org/abs/2409.16797 · HTML: https://mlsanity.com/paper/2409.16797/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 30 of 38)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [ViM: Out-Of-Distribution with Virtual-logit Matching](https://mlsanity.com/paper/2203.10807/) (2022)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space](https://mlsanity.com/paper/2105.01879/) (2021)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection](https://mlsanity.com/paper/2306.09301/) (2023)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Hyperparameter Ensembles for Robustness and Uncertainty Quantification](https://mlsanity.com/paper/2006.13570/) (2020)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [Masksembles for Uncertainty Estimation](https://mlsanity.com/paper/2012.08334/) (2020)
- [A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others](https://mlsanity.com/paper/2212.04825/) (2022)
- [Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization](https://mlsanity.com/paper/2105.05612/) (2021)
- [Agree to Disagree: Diversity through Disagreement for Better Transferability](https://mlsanity.com/paper/2202.04414/) (2022)
- [Neural Ensemble Search for Uncertainty Estimation and Dataset Shift](https://mlsanity.com/paper/2006.08573/) (2020)
- [Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning](https://mlsanity.com/paper/2207.02598/) (2022)
- [Spurious Feature Diversification Improves Out-of-distribution Generalization](https://mlsanity.com/paper/2309.17230/) (2023)
- [Ensembles of Locally Independent Prediction Models](https://mlsanity.com/paper/1911.01291/) (2019)
- [ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms](https://mlsanity.com/paper/2310.01755/) (2023)

## Built on by (most cited first, 2 of 2)

- [Are We Done with Object-Centric Learning?](https://mlsanity.com/paper/2504.07092/) (2025)
- [DISCO: Diversifying Sample Condensation for Efficient Model Evaluation](https://mlsanity.com/paper/2510.07959/) (2025)

## Abstract (arXiv)

Training a diverse ensemble of models has several practical applications such as providing candidates for model selection with better out-of-distribution (OOD) generalization, and enabling the detection of OOD samples via Bayesian principles. An existing approach to diverse ensemble training encourages the models to disagree on provided OOD samples. However, the approach is computationally expensive and it requires well-separated ID and OOD examples, such that it has only been demonstrated in small-scale settings. Method. This work presents a method for Scalable Ensemble Diversification (SED) applicable to large-scale settings (e.g. ImageNet) that does not require OOD samples. Instead, SED identifies hard training samples on the fly and encourages the ensemble members to disagree on these. To improve scaling, we show how to avoid the expensive computations in existing methods of exhaustive pairwise disagreements across models. Results. We evaluate the benefits of diversification with experiments on ImageNet. First, for OOD generalization, we observe large benefits from the diversification in multiple settings including output-space (classical) ensembles and weight-space ensembles (model soups). Second, for OOD detection, we turn the diversity of ensemble hypotheses into a novel uncertainty score estimator that surpasses a large number of OOD detection baselines. Code is available here: https://github.com/AlexanderRubinstein/diverse-universe-public.