Legitimate ground-truth-free metrics for deep uncertainty classification scoring
Not on the map. Why?
Abstract (arXiv)
Despite the increasing demand for safer machine learning practices, the use of Uncertainty Quantification (UQ) methods in production remains limited. This limitation is exacerbated by the challenge of validating UQ methods in absence of UQ ground truth. In classification tasks, when only a usual set of test data is at hand, several authors suggested different metrics that can be computed from such test points while assessing the quality of quantified uncertainties. This paper investigates such metrics and proves that they are theoretically well-behaved and actually tied to some uncertainty ground truth which is easily interpretable in terms of model prediction trustworthiness ranking. Equipped with those new results, and given the applicability of those metrics in the usual supervised paradigm, we argue that our contributions will help promoting a broader use of UQ in deep learning.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 47 references that are in the map. First posted on arXiv 30 Oct 2024.
Builds on
24 of 47 references.
References · 24
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2020Energy-based Out-of-distribution Detectioncited by 1,299
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2017Deep Bayesian Active Learning with Image Datacited by 819
- 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
- 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
- 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
- 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 338
- 2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 280
12 more
- 2019Human uncertainty makes classification more robustcited by 235
- 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 125
- 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 113
- 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 104
- 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
- 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
- 2020Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithmscited by 34
- 2023A framework for benchmarking class-out-of-distribution detection and its application to ImageNetcited by 31
- 2023Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorialcited by 27
- 2023URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescited by 17
- 2020Detecting Misclassification Errors in Neural Networks with a Gaussian Process Modelcited by 8
- 2022On the Calibration of Probabilistic Classifier Setscited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.