# A general framework for ensemble distribution distillation

arXiv: https://arxiv.org/abs/2002.11531 · HTML: https://mlsanity.com/paper/2002.11531/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 16 of 16)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [ELFNet: Evidential Local-global Fusion for Stereo Matching](https://mlsanity.com/paper/2308.00728/) (2023)
- [A Comprehensive Survey on Evidential Deep Learning and Its Applications](https://mlsanity.com/paper/2409.04720/) (2024)
- [Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2303.10917/) (2023)
- [Compensation Learning in Semantic Segmentation](https://mlsanity.com/paper/2304.13428/) (2023)
- [Self-Aware Trajectory Prediction for Safe Autonomous Driving](https://mlsanity.com/paper/2305.09147/) (2023)
- [Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertainties](https://mlsanity.com/paper/2305.10384/) (2023)
- [TEDL: A Two-stage Evidential Deep Learning Method for Classification Uncertainty Quantification](https://mlsanity.com/paper/2209.05522/) (2022)
- [Accuracy-Preserving Calibration via Statistical Modeling on Probability Simplex](https://mlsanity.com/paper/2402.13765/) (2024)
- [Simple Regularisation for Uncertainty-Aware Knowledge Distillation](https://mlsanity.com/paper/2205.09526/) (2022)
- [Uncertainty Quantification for Image-based Traffic Prediction across Cities](https://mlsanity.com/paper/2308.06129/) (2023)
- [Reliability Quantification of Deep Reinforcement Learning-based Control](https://mlsanity.com/paper/2309.16977/) (2023)
- [Gaussian Mixture based Evidential Learning for Stereo Matching](https://mlsanity.com/paper/2408.02796/) (2024)
- [Ensemble Distribution Distillation for Self-Supervised Human Activity Recognition](https://mlsanity.com/paper/2509.08225/) (2025)
- [Probabilistic Machine Learning for Uncertainty-Aware Diagnosis of Industrial Systems](https://mlsanity.com/paper/2509.18810/) (2025)

## Abstract (arXiv)

Ensembles of neural networks have been shown to give better performance than single networks, both in terms of predictions and uncertainty estimation. Additionally, ensembles allow the uncertainty to be decomposed into aleatoric (data) and epistemic (model) components, giving a more complete picture of the predictive uncertainty. Ensemble distillation is the process of compressing an ensemble into a single model, often resulting in a leaner model that still outperforms the individual ensemble members. Unfortunately, standard distillation erases the natural uncertainty decomposition of the ensemble. We present a general framework for distilling both regression and classification ensembles in a way that preserves the decomposition. We demonstrate the desired behaviour of our framework and show that its predictive performance is on par with standard distillation.