mlsanityData to 7 Oct 2026

Improving robustness and calibration in ensembles with diversity regularization

arXiv 2201.10908 ↗First posted 26 Jan 2022cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Calibration and uncertainty estimation are crucial topics in high-risk environments. We introduce a new diversity regularizer for classification tasks that uses out-of-distribution samples and increases the overall accuracy, calibration and out-of-distribution detection capabilities of ensembles. Following the recent interest in the diversity of ensembles, we systematically evaluate the viability of explicitly regularizing ensemble diversity to improve calibration on in-distribution data as well as under dataset shift. We demonstrate that diversity regularization is highly beneficial in architectures, where weights are partially shared between the individual members and even allows to use fewer ensemble members to reach the same level of robustness. Experiments on CIFAR-10, CIFAR-100, and SVHN show that regularizing diversity can have a significant impact on calibration and robustness, as well as out-of-distribution detection.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

36 of 45 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  11. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  12. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
24 more
  1. 2015Weight Uncertainty in Neural Networkscited by 554
  2. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  3. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  4. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  5. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  6. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  7. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  8. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  9. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  10. 2020Training independent subnetworks for robust predictioncited by 177
  11. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  12. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 138
  13. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  14. 2018Measuring and regularizing networks in function spacecited by 97
  15. 2019Improving Adversarial Robustness of Ensembles with Diversity Trainingcited by 69
  16. 2020Depth Uncertainty in Neural Networkscited by 56
  17. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  18. 2020Hydra: Preserving Ensemble Diversity for Model Distillationcited by 31
  19. 2019Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensemblescited by 30
  20. 2020Diversity inducing Information Bottleneck in Model Ensemblescited by 27
  21. 2020Diverse Ensembles Improve Calibrationcited by 23
  22. 2016Swapout: Learning an ensemble of deep architecturescited by 19
  23. 2018Diversity regularization in deep ensemblescited by 11
  24. 2020Dropout Strikes Back: Improved Uncertainty Estimation via Diversity Samplingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 0; 2026: 1.

8 papers.

  1. 2023Joint Training of Deep Ensembles Fails Due to Learner Collusioncited by 23
  2. 2023Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantificationcited by 6
  3. 2022Bayesian posterior approximation with stochastic ensemblescited by 5
  4. 2024Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentationcited by 5
  5. 2022Simple Regularisation for Uncertainty-Aware Knowledge Distillationcited by 1
  6. 2024Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentationcited by 1
  7. 2024Uncertainty quantification for White Matter Hyperintensity segmentation detects silent failures and improves automated Fazekas quantificationcited by 0
  8. 2026Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulationcited by 0