# On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation

arXiv: https://arxiv.org/abs/1806.02562 · HTML: https://mlsanity.com/paper/1806.02562/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2018.

## Built on by (most cited first, 30 of 46)

- [Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining](https://mlsanity.com/paper/2402.03302/) (2024)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [A Review of Uncertainty Estimation and its Application in Medical Imaging](https://mlsanity.com/paper/2302.08119/) (2023)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations](https://mlsanity.com/paper/2104.05788/) (2021)
- [TBraTS: Trusted Brain Tumor Segmentation](https://mlsanity.com/paper/2206.09309/) (2022)
- [Towards Lower-Dose PET using Physics-Based Uncertainty-Aware Multimodal Learning with Robustness to Out-of-Distribution Data](https://mlsanity.com/paper/2107.09892/) (2021)
- [FUTURE-AI: Guiding Principles and Consensus Recommendations for Trustworthy Artificial Intelligence in Medical Imaging](https://mlsanity.com/paper/2109.09658/) (2021)
- [Label fusion and training methods for reliable representation of inter-rater uncertainty](https://mlsanity.com/paper/2202.07550/) (2022)
- [Modeling Annotator Preference and Stochastic Annotation Error for Medical Image Segmentation](https://mlsanity.com/paper/2111.13410/) (2021)
- [Generalized Probabilistic U-Net for medical image segementation](https://mlsanity.com/paper/2207.12872/) (2022)
- [Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation](https://mlsanity.com/paper/2210.17398/) (2022)
- [A data-centric approach for improving ambiguous labels with combined semi-supervised classification and clustering](https://mlsanity.com/paper/2106.16209/) (2021)
- [A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation](https://mlsanity.com/paper/2109.10702/) (2021)
- [Diversified and Personalized Multi-rater Medical Image Segmentation](https://mlsanity.com/paper/2403.13417/) (2024)
- [Impact of individual rater style on deep learning uncertainty in medical imaging segmentation](https://mlsanity.com/paper/2105.02197/) (2021)
- [Transformer-based Annotation Bias-aware Medical Image Segmentation](https://mlsanity.com/paper/2306.01340/) (2023)
- [Effect of latent space distribution on the segmentation of images with multiple annotations](https://mlsanity.com/paper/2304.13476/) (2023)
- [Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results](https://mlsanity.com/paper/2505.08685/) (2025)
- [Modeling Annotation Uncertainty with Gaussian Heatmaps in Landmark Localization](https://mlsanity.com/paper/2109.09533/) (2021)
- [Automatic Segmentation of Gross Target Volume of Nasopharynx Cancer using Ensemble of Multiscale Deep Neural Networks with Spatial Attention](https://mlsanity.com/paper/2101.11254/) (2021)
- [MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering](https://mlsanity.com/paper/2107.03216/) (2021)
- [Opinions Vary? Diagnosis First!](https://mlsanity.com/paper/2202.06505/) (2022)
- [Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle](https://mlsanity.com/paper/2208.03016/) (2022)
- [USE-Evaluator: Performance Metrics for Medical Image Segmentation Models with Uncertain, Small or Empty Reference Annotations](https://mlsanity.com/paper/2209.13008/) (2022)
- [Quantifying U-Net Uncertainty in Multi-Parametric MRI-based Glioma Segmentation by Spherical Image Projection](https://mlsanity.com/paper/2210.06512/) (2022)
- [Multi-rater Prompting for Ambiguous Medical Image Segmentation](https://mlsanity.com/paper/2404.07580/) (2024)
- [Bridging the Perception-Cognition Gap:Re-engineering SAM2 with Hilbert-Mamba for Robust VLM-based Medical Diagnosis](https://mlsanity.com/paper/2512.24013/) (2025)
- [Cohort Bias Adaptation in Aggregated Datasets for Lesion Segmentation](https://mlsanity.com/paper/2108.00713/) (2021)

## Abstract (arXiv)

Uncertainty estimation methods are expected to improve the understanding and quality of computer-assisted methods used in medical applications (e.g., neurosurgical interventions, radiotherapy planning), where automated medical image segmentation is crucial. In supervised machine learning, a common practice to generate ground truth label data is to merge observer annotations. However, as many medical image tasks show a high inter-observer variability resulting from factors such as image quality, different levels of user expertise and domain knowledge, little is known as to how inter-observer variability and commonly used fusion methods affect the estimation of uncertainty of automated image segmentation. In this paper we analyze the effect of common image label fusion techniques on uncertainty estimation, and propose to learn the uncertainty among observers. The results highlight the negative effect of fusion methods applied in deep learning, to obtain reliable estimates of segmentation uncertainty. Additionally, we show that the learned observers' uncertainty can be combined with current standard Monte Carlo dropout Bayesian neural networks to characterize uncertainty of model's parameters.