# Improving robustness and calibration in ensembles with diversity regularization

arXiv: https://arxiv.org/abs/2201.10908 · HTML: https://mlsanity.com/paper/2201.10908/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Improving Adversarial Robustness via Promoting Ensemble Diversity](https://mlsanity.com/paper/1901.08846/) (2019)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Hyperparameter Ensembles for Robustness and Uncertainty Quantification](https://mlsanity.com/paper/2006.13570/) (2020)
- [Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks](https://mlsanity.com/paper/1511.06314/) (2015)
- [Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness](https://mlsanity.com/paper/1905.13472/) (2019)
- [Measuring and regularizing networks in function space](https://mlsanity.com/paper/1805.08289/) (2018)
- [Improving Adversarial Robustness of Ensembles with Diversity Training](https://mlsanity.com/paper/1901.09981/) (2019)
- [Depth Uncertainty in Neural Networks](https://mlsanity.com/paper/2006.08437/) (2020)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Hydra: Preserving Ensemble Diversity for Model Distillation](https://mlsanity.com/paper/2001.04694/) (2020)

## Built on by (most cited first, 8 of 8)

- [Joint Training of Deep Ensembles Fails Due to Learner Collusion](https://mlsanity.com/paper/2301.11323/) (2023)
- [Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification](https://mlsanity.com/paper/2304.04042/) (2023)
- [Bayesian posterior approximation with stochastic ensembles](https://mlsanity.com/paper/2212.08123/) (2022)
- [Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation](https://mlsanity.com/paper/2412.11608/) (2024)
- [Simple Regularisation for Uncertainty-Aware Knowledge Distillation](https://mlsanity.com/paper/2205.09526/) (2022)
- [Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentation](https://mlsanity.com/paper/2407.14306/) (2024)
- [Uncertainty quantification for White Matter Hyperintensity segmentation detects silent failures and improves automated Fazekas quantification](https://mlsanity.com/paper/2411.17571/) (2024)
- [Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation](https://mlsanity.com/paper/2607.23860/) (2026)

## Abstract (arXiv)

Calibration and uncertainty estimation are crucial topics in high-risk environments. We introduce a new diversity regularizer for classification tasks that uses out-of-distribution samples and increases the overall accuracy, calibration and out-of-distribution detection capabilities of ensembles. Following the recent interest in the diversity of ensembles, we systematically evaluate the viability of explicitly regularizing ensemble diversity to improve calibration on in-distribution data as well as under dataset shift. We demonstrate that diversity regularization is highly beneficial in architectures, where weights are partially shared between the individual members and even allows to use fewer ensemble members to reach the same level of robustness. Experiments on CIFAR-10, CIFAR-100, and SVHN show that regularizing diversity can have a significant impact on calibration and robustness, as well as out-of-distribution detection.