# Label fusion and training methods for reliable representation of inter-rater uncertainty

arXiv: https://arxiv.org/abs/2202.07550 · HTML: https://mlsanity.com/paper/2202.07550/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 23 of 23)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation](https://mlsanity.com/paper/2006.16806/) (2020)
- [Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge](https://mlsanity.com/paper/1802.10508/) (2018)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [Bayesian QuickNAT: Model Uncertainty in Deep Whole-Brain Segmentation for Structure-wise Quality Control](https://mlsanity.com/paper/1811.09800/) (2018)
- [Simulation of hyperelastic materials in real-time using Deep Learning](https://mlsanity.com/paper/1904.06197/) (2019)
- [SoftSeg: Advantages of soft versus binary training for image segmentation](https://mlsanity.com/paper/2011.09041/) (2020)
- [Using Soft Labels to Model Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/2109.12622/) (2021)
- [Difficulty-aware Glaucoma Classification with Multi-Rater Consensus Modeling](https://mlsanity.com/paper/2007.14848/) (2020)
- [D-LEMA: Deep Learning Ensembles from Multiple Annotations -- Application to Skin Lesion Segmentation](https://mlsanity.com/paper/2012.07206/) (2020)
- [Automatic segmentation of the spinal cord and intramedullary multiple sclerosis lesions with convolutional neural networks](https://mlsanity.com/paper/1805.06349/) (2018)
- [ivadomed: A Medical Imaging Deep Learning Toolbox](https://mlsanity.com/paper/2010.09984/) (2020)
- [Impact of individual rater style on deep learning uncertainty in medical imaging segmentation](https://mlsanity.com/paper/2105.02197/) (2021)
- [A Soft STAPLE Algorithm Combined with Anatomical Knowledge](https://mlsanity.com/paper/1910.12077/) (2019)
- [Benefits of Linear Conditioning with Metadata for Image Segmentation](https://mlsanity.com/paper/2102.09582/) (2021)
- [Optimizing Operating Points for High Performance Lesion Detection and Segmentation Using Lesion Size Reweighting](https://mlsanity.com/paper/2107.12978/) (2021)

## Built on by (most cited first, 13 of 13)

- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [On Image Segmentation With Noisy Labels: Characterization and Volume Properties of the Optimal Solutions to Accuracy and Dice](https://mlsanity.com/paper/2206.06484/) (2022)
- [Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels](https://mlsanity.com/paper/2303.16296/) (2023)
- [How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation](https://mlsanity.com/paper/2308.06964/) (2023)
- [Noisy Image Segmentation With Soft-Dice](https://mlsanity.com/paper/2304.00801/) (2023)
- [Multi-Domain Data Aggregation for Axon and Myelin Segmentation in Histology Images](https://mlsanity.com/paper/2409.11552/) (2024)
- [Reliability of deep learning models for anatomical landmark detection: The role of inter-rater variability](https://mlsanity.com/paper/2411.17850/) (2024)
- [Marginal Thresholding in Noisy Image Segmentation](https://mlsanity.com/paper/2304.04116/) (2023)
- [Monitoring morphometric drift in lifelong learning segmentation of the spinal cord](https://mlsanity.com/paper/2505.01364/) (2025)
- [Semi-Supervised Multi-Task Learning for Interpretable Quality As- sessment of Fundus Images](https://mlsanity.com/paper/2511.13353/) (2025)
- [Leveraging Multi-Rater Annotations to Calibrate Object Detectors in Microscopy Imaging](https://mlsanity.com/paper/2601.23007/) (2026)
- [Multi-Rater Calibrated Segmentation Models](https://mlsanity.com/paper/2605.02437/) (2026)

## Abstract (arXiv)

Medical tasks are prone to inter-rater variability due to multiple factors such as image quality, professional experience and training, or guideline clarity. Training deep learning networks with annotations from multiple raters is a common practice that mitigates the model's bias towards a single expert. Reliable models generating calibrated outputs and reflecting the inter-rater disagreement are key to the integration of artificial intelligence in clinical practice. Various methods exist to take into account different expert labels. We focus on comparing three label fusion methods: STAPLE, average of the rater's segmentation, and random sampling of each rater's segmentation during training. Each label fusion method is studied using both the conventional training framework and the recently published SoftSeg framework that limits information loss by treating the segmentation task as a regression. Our results, across 10 data splittings on two public datasets, indicate that SoftSeg models, regardless of the ground truth fusion method, had better calibration and preservation of the inter-rater rater variability compared with their conventional counterparts without impacting the segmentation performance. Conventional models, i.e., trained with a Dice loss, with binary inputs, and sigmoid/softmax final activate, were overconfident and underestimated the uncertainty associated with inter-rater variability. Conversely, fusing labels by averaging with the SoftSeg framework led to underconfident outputs and overestimation of the rater disagreement. In terms of segmentation performance, the best label fusion method was different for the two datasets studied, indicating this parameter might be task-dependent. However, SoftSeg had segmentation performance systematically superior or equal to the conventionally trained models and had the best calibration and preservation of the inter-rater variability.