mlsanityData to 14 Oct 2026

On the Calibration of Probabilistic Classifier Sets

arXiv 2205.10082 ↗First posted 20 May 2022stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Multi-class classification methods that produce sets of probabilistic classifiers, such as ensemble learning methods, are able to model aleatoric and epistemic uncertainty. Aleatoric uncertainty is then typically quantified via the Bayes error, and epistemic uncertainty via the size of the set. In this paper, we extend the notion of calibration, which is commonly used to evaluate the validity of the aleatoric uncertainty representation of a single probabilistic classifier, to assess the validity of an epistemic uncertainty representation obtained by sets of probabilistic classifiers. Broadly speaking, we call a set of probabilistic classifiers calibrated if one can find a calibrated convex combination of these classifiers. To evaluate this notion of calibration, we propose a novel nonparametric calibration test that generalizes an existing test for single probabilistic classifiers to the case of sets of probabilistic classifiers. Making use of this test, we empirically show that ensembles of deep neural networks are often not well calibrated.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 17 of its 47 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

17 of 47 references.

References · 17

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  7. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  8. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  9. 2019Evaluating model calibration in classificationcited by 168
  10. 2020Uncertainty Quantification and Deep Ensemblescited by 139
  11. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  12. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
5 more
  1. 2020Uncertainty in Gradient Boosting via Ensemblescited by 56
  2. 2020Aleatoric and Epistemic Uncertainty with Random Forestscited by 43
  3. 2020Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable?cited by 39
  4. 2021Valid prediction intervals for regression problemscited by 29
  5. 2022Aggregating distribution forecasts from deep ensemblescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 1.

8 papers.

  1. 2024Credal Learning Theorycited by 31
  2. 2024Conformalized Credal Regions for Classification with Ambiguous Ground Truthcited by 19
  3. 2025Conformal Prediction Regions are Imprecise Highest Density Regionscited by 7
  4. 2025A calibration test for evaluating set-based epistemic uncertainty representationscited by 7
  5. 2024Legitimate ground-truth-free metrics for deep uncertainty classification scoringcited by 1
  6. 2025Robust Partial-Label Learning by Leveraging Class Activation Valuescited by 1
  7. 2024Optimal Transport for ε-Contaminated Credal Sets: To the Memory of Sayan Mukherjeecited by 0
  8. 2026Calibrated Preference Learning: The Case of Label Rankingcited by 0