mlsanityData to 7 Oct 2026

(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Models

arXiv 2409.02628 ↗First posted 4 Sep 2024cs.LG stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

Epistemic uncertainty is crucial for safety-critical applications and data acquisition tasks. Yet, we find an important phenomenon in deep learning models: an epistemic uncertainty collapse as model complexity increases, challenging the assumption that larger models invariably offer better uncertainty quantification. We introduce implicit ensembling as a possible explanation for this phenomenon. To investigate this hypothesis, we provide theoretical analysis and experiments that demonstrate uncertainty collapse in explicit ensembles of ensembles and show experimental evidence of similar collapse in wider models across various architectures, from simple MLPs to state-of-the-art vision models including ResNets and Vision Transformers. We further develop implicit ensemble extraction techniques to decompose larger models into diverse sub-models, showing we can thus recover epistemic uncertainty. We explore the implications of these findings for uncertainty estimation.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 41 of its 61 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

41 of 61 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2016Wide Residual Networkscited by 3,057
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  11. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
29 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  3. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  4. 2017Deep Bayesian Active Learning with Image Datacited by 812
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  7. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  8. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  9. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  10. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  11. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  12. 2020Supermasks in Superpositioncited by 223
  13. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  14. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  15. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  16. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  17. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  18. 2023Modern Bayesian Experimental Designcited by 121
  19. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  20. 2020Masksembles for Uncertainty Estimationcited by 88
  21. 2022Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisationcited by 65
  22. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  23. 2022Diversify and Disambiguate: Learning From Underspecified Datacited by 55
  24. 2022Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Taskscited by 49
  25. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 28
  26. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  27. 2024Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage?cited by 17
  28. 2023Black-Box Batch Active Learning for Regressioncited by 12
  29. 2024The Epistemic Uncertainty Hole: an issue of Bayesian Neural Networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
  2. 2026Agentic Uncertainty Quantificationcited by 16
  3. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  4. 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
  5. 2026Position: agentic AI orchestration should be Bayes-consistentcited by 3
  6. 2025From predictions to confidence intervals: an empirical study of conformal prediction methods for in-context learningcited by 2
  7. 2025Uncertainty Estimation using Variance-Gated Distributionscited by 2
  8. 2025Scaling Laws for Uncertainty in Deep Learningcited by 1
  9. 2026Rethinking Uncertainty Quantification and Entanglement in Image Segmentationcited by 1
  10. 2025Entropy-regularized Gradient Estimators for Approximate Bayesian Inferencecited by 0
  11. 2026Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?cited by 0
  12. 2026Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantificationcited by 0
3 more
  1. 2026Uncertainty quantification for trustworthy deep learning: Methods and measurescited by 0
  2. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  3. 2026Structural Limits of the Information-Theoretic Uncertainty Decompositioncited by 0