# Diversity Matters When Learning From Ensembles

arXiv: https://arxiv.org/abs/2110.14149 · HTML: https://mlsanity.com/paper/2110.14149/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Sobolev Training for Neural Networks](https://mlsanity.com/paper/1706.04859/) (2017)
- [Knowledge Transfer with Jacobian Matching](https://mlsanity.com/paper/1803.00443/) (2018)
- [Hydra: Preserving Ensemble Diversity for Model Distillation](https://mlsanity.com/paper/2001.04694/) (2020)
- [Early Stopping is Nonparametric Variational Inference](https://mlsanity.com/paper/1504.01344/) (2015)

## Built on by (most cited first, 27 of 27)

- [Deep Isolation Forest for Anomaly Detection](https://mlsanity.com/paper/2206.06602/) (2022)
- [Accelerating Dataset Distillation via Model Augmentation](https://mlsanity.com/paper/2212.06152/) (2022)
- [Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping](https://mlsanity.com/paper/2206.03633/) (2022)
- [Verification-Aided Deep Ensemble Selection](https://mlsanity.com/paper/2202.03898/) (2022)
- [Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation](https://mlsanity.com/paper/2206.15047/) (2022)
- [Memory-adaptive Depth-wise Heterogeneous Federated Learning](https://mlsanity.com/paper/2303.04887/) (2023)
- [MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2405.18897/) (2024)
- [Bayesian posterior approximation with stochastic ensembles](https://mlsanity.com/paper/2212.08123/) (2022)
- [Functional Ensemble Distillation](https://mlsanity.com/paper/2206.02183/) (2022)
- [Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applications](https://mlsanity.com/paper/2302.01440/) (2023)
- [Medical Image Debiasing by Learning Adaptive Agreement from a Biased Council](https://mlsanity.com/paper/2401.11713/) (2024)
- [Fast Ensembling with Diffusion Schr"odinger Bridge](https://mlsanity.com/paper/2404.15814/) (2024)
- [Fairness of Deep Ensembles: On the interplay between per-group task difficulty and under-representation](https://mlsanity.com/paper/2501.14551/) (2025)
- [Unsupervised Machine Learning for Explainable Health Care Fraud Detection](https://mlsanity.com/paper/2211.02927/) (2022)
- [Exploring Resiliency to Natural Image Corruptions in Deep Learning using Design Diversity](https://mlsanity.com/paper/2303.09283/) (2023)
- [AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectories](https://mlsanity.com/paper/2310.10541/) (2023)
- [Towards a Probabilistic Fusion Approach for Robust Battery Prognostics](https://mlsanity.com/paper/2405.15292/) (2024)
- [Deep Nets with Subsampling Layers Unwittingly Discard Useful Activations at Test-Time](https://mlsanity.com/paper/2410.01083/) (2024)
- [Knowledge Distillation of Uncertainty using Deep Latent Factor Model](https://mlsanity.com/paper/2510.19290/) (2025)
- [Distilling Influences to Mitigate Prediction Churn in Graph Neural Networks](https://mlsanity.com/paper/2310.00946/) (2023)
- [Efficient Multi-Model Fusion with Adversarial Complementary Representation Learning](https://mlsanity.com/paper/2404.15704/) (2024)
- [Comparison of Deterministic and Probabilistic Machine Learning Algorithms for Precise Dimensional Control and Uncertainty Quantification in Additive Manufacturing](https://mlsanity.com/paper/2509.16233/) (2025)
- [MME: Mixture of Mesh Experts with Random Walk Transformer Gating](https://mlsanity.com/paper/2603.00828/) (2026)
- [LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning](https://mlsanity.com/paper/2605.14365/) (2026)
- [DAMEL: Dual-Axis Multi-Expert Learning for Class-Imbalanced Learning](https://mlsanity.com/paper/2605.30135/) (2026)
- [Efficient Bayesian Deep Ensembles via Analytic Predictive Inference](https://mlsanity.com/paper/2607.06776/) (2026)
- [The Distributional View of Knowledge Distillation](https://mlsanity.com/paper/2608.15215/) (2026)

## Abstract (arXiv)

Deep ensembles excel in large-scale image classification tasks both in terms of prediction accuracy and calibration. Despite being simple to train, the computation and memory cost of deep ensembles limits their practicability. While some recent works propose to distill an ensemble model into a single model to reduce such costs, there is still a performance gap between the ensemble and distilled models. We propose a simple approach for reducing this gap, i.e., making the distilled performance close to the full ensemble. Our key assumption is that a distilled model should absorb as much function diversity inside the ensemble as possible. We first empirically show that the typical distillation procedure does not effectively transfer such diversity, especially for complex models that achieve near-zero training error. To fix this, we propose a perturbation strategy for distillation that reveals diversity by seeking inputs for which ensemble member outputs disagree. We empirically show that a model distilled with such perturbed samples indeed exhibits enhanced diversity, leading to improved performance.