mlsanityData to 7 Oct 2026

Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning

arXiv 2002.06470 ↗First posted 15 Feb 2020stat.ML cs.LGcited by 208

Not on the map. Why?

Abstract (arXiv)

Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of only few independently trained networks in terms of test performance.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 46; 2023: 50; 2024: 36; 2025: 16; 2026: 12.

208 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2022Continual Test-Time Domain Adaptationcited by 520
  3. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  4. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  5. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  6. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  7. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  8. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  9. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  10. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  11. 2022A high-resolution canopy height model of the Earthcited by 91
  12. 2023DELTA: degradation-free fully test-time adaptationcited by 72
38 more
  1. 2022Conditional-Flow NeRF: Accurate 3D Modelling with Reliable Uncertainty Quantificationcited by 71
  2. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  3. 2022Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysiscited by 62
  4. 2023A Review of Uncertainty Estimation and its Application in Medical Imagingcited by 59
  5. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  6. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  7. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  8. 2021Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibrationcited by 48
  9. 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 44
  10. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  11. 2021Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspectivecited by 39
  12. 2022LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinitycited by 39
  13. 2021Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectorscited by 37
  14. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  15. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  16. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  17. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  18. 2021AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantificationcited by 31
  19. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 31
  20. 2022Calibrated Selective Classificationcited by 30
  21. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  22. 2021QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Resultscited by 28
  23. 2022Conformal Prediction Sets with Limited False Positivescited by 28
  24. 2023TeSLA: Test-Time Self-Learning With Automatic Adversarial Augmentationcited by 28
  25. 2023Quantification of Uncertainty with Adversarial Modelscited by 28
  26. 2021Deep Ensembles from a Bayesian Perspectivecited by 27
  27. 2021Diversity Matters When Learning From Ensemblescited by 27
  28. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  29. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  30. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  31. 2022Pre-training General Trajectory Embeddings with Maximum Multi-view Entropy Codingcited by 25
  32. 2021Leveraging Self-Supervision for Cross-Domain Crowd Countingcited by 24
  33. 2022MUAD: Multiple Uncertainties for Autonomous Driving, a benchmark for multiple uncertainty types and taskscited by 24
  34. 2022FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulationcited by 24
  35. 2024Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Taskscited by 24
  36. 2021Robustness via Cross-Domain Ensemblescited by 23
  37. 2021Diversity Enhanced Active Learning with Strictly Proper Scoring Rulescited by 22
  38. 2022Shifts 2.0: Extending The Dataset of Real Distributional Shiftscited by 22

50 most cited of 208 All 208 (JSON)