# Multi-Rater Calibrated Segmentation Models

arXiv: https://arxiv.org/abs/2605.02437 · HTML: https://mlsanity.com/paper/2605.02437/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 4 May 2026.

## Builds on (most cited first, 22 of 22)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [To Smooth or Not? When Label Smoothing Meets Noisy Labels](https://mlsanity.com/paper/2106.04149/) (2021)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations](https://mlsanity.com/paper/2104.05788/) (2021)
- [On Calibrating Semantic Segmentation Models: Analyses and An Algorithm](https://mlsanity.com/paper/2212.12053/) (2022)
- [Using Soft Labels to Model Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/2109.12622/) (2021)
- [Label fusion and training methods for reliable representation of inter-rater uncertainty](https://mlsanity.com/paper/2202.07550/) (2022)
- [Improving the repeatability of deep learning models with Monte Carlo dropout](https://mlsanity.com/paper/2202.07562/) (2022)
- [Cost-Sensitive Regularization for Diabetic Retinopathy Grading from Eye Fundus Images](https://mlsanity.com/paper/2010.00291/) (2020)
- [Decompositions of the mean continuous ranked probability score](https://mlsanity.com/paper/2311.14122/) (2023)
- [Multi-Head Multi-Loss Model Calibration](https://mlsanity.com/paper/2303.01099/) (2023)
- [Uncertainty Aware Training to Improve Deep Learning Model Calibration for Classification of Cardiac MR Images](https://mlsanity.com/paper/2308.15141/) (2023)
- [Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results](https://mlsanity.com/paper/2505.08685/) (2025)
- [Addressing Deep Learning Model Calibration Using Evidential Neural Networks and Uncertainty-Aware Training](https://mlsanity.com/paper/2301.13296/) (2023)
- [Performance Metrics for Probabilistic Ordinal Classifiers](https://mlsanity.com/paper/2309.08701/) (2023)
- [Learning Robust Medical Image Segmentation from Multi-source Annotations](https://mlsanity.com/paper/2304.00466/) (2023)
- [DIOR-ViT: Differential Ordinal Learning Vision Transformer for Cancer Classification in Pathology Images](https://mlsanity.com/paper/2407.08503/) (2024)

## Abstract (arXiv)

Objective: Accurate probability estimates are essential for the safe deployment of medical image segmentation models in clinical decision-making. However, modern deep segmentation networks are often poorly calibrated, a problem exacerbated when multiple expert annotations exhibit substantial disagreement. While inter-rater variability is typically treated as noise, it provides valuable information about intrinsic annotation ambiguity that must be reflected in model confidence. Methods: We improve the probabilistic calibration of medical image segmentation models by reformulating multi-rater supervision as an ordinal learning problem. Voxel-wise annotator agreement is treated as an ordered target, linking predictive confidence to the empirical variability in training data. This formulation allows the use of ordinal-aware scoring rules, such as the Ranked Probability Score ordinal loss, combined with a standard binary objective to preserve discriminative performance. Results: We evaluated the proposed approach across four public segmentation benchmarks spanning ophthalmology, histopathology, and thoracic imaging. Calibration was assessed using a multi-rater extension of expected calibration error. Results consistently show that ordinal-aware training yields substantially improved calibration with respect to inter-rater agreement without degrading segmentation accuracy. Conclusions: Treating multi-rater annotations as ordered information provides a principled and architecture-agnostic route to more reliable probabilistic segmentation models.