mlsanityData to 7 Oct 2026

Self-Distribution Distillation: Efficient Uncertainty Estimation

arXiv 2203.08295 ↗First posted 15 Mar 2022cs.LG cs.AI stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

Deep learning is increasingly being applied in safety-critical domains. For these scenarios it is important to know the level of uncertainty in a model's prediction to ensure appropriate decisions are made by the system. Deep ensembles are the de-facto standard approach to obtaining various measures of uncertainty. However, ensembles often significantly increase the resources required in the training and/or deployment phases. Approaches have been developed that typically address the costs in one of these phases. In this work we propose a novel training approach, self-distribution distillation (S2D), which is able to efficiently train a single model that can estimate uncertainties. Furthermore it is possible to build ensembles of these models and apply hierarchical ensemble distillation approaches. Experiments on CIFAR-100 showed that S2D models outperformed standard models and Monte-Carlo dropout. Additional out-of-distribution detection experiments on LSUN, Tiny ImageNet, SVHN showed that even a standard deep ensemble can be outperformed using S2D based ensembles and novel distilled models.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

31 of 48 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2016Wide Residual Networkscited by 3,057
  9. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
19 more
  1. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  2. 2016Deep Networks with Stochastic Depthcited by 973
  3. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  4. 2016Sequence-Level Knowledge Distillationcited by 761
  5. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  6. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  7. 2015Weight Uncertainty in Neural Networkscited by 554
  8. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  9. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  10. 2018Learning to Drive in a Daycited by 266
  11. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  12. 2020Training independent subnetworks for robust predictioncited by 177
  13. 2019Ensemble Distribution Distillationcited by 158
  14. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  15. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  16. 2018Learning for Single-Shot Confidence Calibration in Deep Neural Networks through Stochastic Inferencescited by 25
  17. 2021Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targetscited by 19
  18. 2020Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networkscited by 14
  19. 2020Ensemble Distillation Approaches for Grammatical Error Correctioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 3; 2025: 1; 2026: 3.

11 papers.

  1. 2024Uncertainty Estimation by Density Aware Evidential Deep Learningcited by 22
  2. 2022Hyperbolic Uncertainty Aware Semantic Segmentationcited by 17
  3. 2024Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage?cited by 17
  4. 2023Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Modelscited by 15
  5. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  6. 2023Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertaintiescited by 5
  7. 2026Semantic Self-Distillation for Language Model Uncertaintycited by 4
  8. 2023Who Needs Decoders? Efficient Estimation of Sequence-level Attributescited by 3
  9. 2025BotUmc: An Uncertainty-Aware Twitter Bot Detection with Multi-view Causal Inferencecited by 0
  10. 2026Density-Informed Pseudo-Counts for Calibrated Evidential Deep Learningcited by 0
  11. 2026Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Yearcited by 0