# Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain

arXiv: https://arxiv.org/abs/2007.01659 · HTML: https://mlsanity.com/paper/2007.01659/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 7 of 7)

- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [Towards Calibrated Deep Clustering Network](https://mlsanity.com/paper/2403.02998/) (2024)
- [Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models](https://mlsanity.com/paper/2411.03497/) (2024)
- [Perception and Semantic Aware Regularization for Sequential Confidence Calibration](https://mlsanity.com/paper/2305.19498/) (2023)
- [AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment](https://mlsanity.com/paper/2605.11398/) (2026)
- [ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty](https://mlsanity.com/paper/2610.17016/) (2026)
- [Noisy-Label Calibration with Multiple Unreliable Labels](https://mlsanity.com/paper/2610.19543/) (2026)

## Abstract (arXiv)

We propose an evaluation framework for class probability estimates (CPEs) in the presence of label uncertainty, which is commonly observed as diagnosis disagreement between experts in the medical domain. We also formalize evaluation metrics for higher-order statistics, including inter-rater disagreement, to assess predictions on label uncertainty. Moreover, we propose a novel post-hoc method called alpha-calibration, that equips neural network classifiers with calibrated distributions over CPEs. Using synthetic experiments and a large-scale medical imaging application, we show that our approach significantly enhances the reliability of uncertainty estimates: disagreement probabilities and posterior CPEs.