# Impact of individual rater style on deep learning uncertainty in medical imaging segmentation

arXiv: https://arxiv.org/abs/2105.02197 · HTML: https://mlsanity.com/paper/2105.02197/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 17 of 17)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [Disentangling Human Error from the Ground Truth in Segmentation of Medical Images](https://mlsanity.com/paper/2007.15963/) (2020)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [SoftSeg: Advantages of soft versus binary training for image segmentation](https://mlsanity.com/paper/2011.09041/) (2020)
- [Let's agree to disagree: learning highly debatable multirater labelling](https://mlsanity.com/paper/1909.01891/) (2019)
- [Automatic segmentation of the spinal cord and intramedullary multiple sclerosis lesions with convolutional neural networks](https://mlsanity.com/paper/1805.06349/) (2018)
- [ivadomed: A Medical Imaging Deep Learning Toolbox](https://mlsanity.com/paper/2010.09984/) (2020)
- [Benefits of Linear Conditioning with Metadata for Image Segmentation](https://mlsanity.com/paper/2102.09582/) (2021)
- [Automatic segmentation of spinal multiple sclerosis lesions: How to generalize across MRI contrasts?](https://mlsanity.com/paper/2003.04377/) (2020)
- [The Worrisome Impact of an Inter-rater Bias on Neural Network Training](https://mlsanity.com/paper/1906.11872/) (2019)

## Built on by (most cited first, 8 of 8)

- [Label fusion and training methods for reliable representation of inter-rater uncertainty](https://mlsanity.com/paper/2202.07550/) (2022)
- [Modeling Annotator Preference and Stochastic Annotation Error for Medical Image Segmentation](https://mlsanity.com/paper/2111.13410/) (2021)
- [Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation](https://mlsanity.com/paper/2210.17398/) (2022)
- [How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation](https://mlsanity.com/paper/2308.06964/) (2023)
- [Cohort Bias Adaptation in Aggregated Datasets for Lesion Segmentation](https://mlsanity.com/paper/2108.00713/) (2021)
- [Random Expert Sampling for Deep Learning Segmentation of Acute Ischemic Stroke on Non-contrast CT](https://mlsanity.com/paper/2309.03930/) (2023)
- [Ensemble CNN and Uncertainty Modeling to Improve Automatic Identification/Segmentation of Multiple Sclerosis Lesions in Magnetic Resonance Imaging](https://mlsanity.com/paper/2108.11791/) (2021)
- [Generating 3D Bio-Printable Patches Using Wound Segmentation and Reconstruction to Treat Diabetic Foot Ulcers](https://mlsanity.com/paper/2203.03814/) (2022)

## Abstract (arXiv)

While multiple studies have explored the relation between inter-rater variability and deep learning model uncertainty in medical segmentation tasks, little is known about the impact of individual rater style. This study quantifies rater style in the form of bias and consistency and explores their impacts when used to train deep learning models. Two multi-rater public datasets were used, consisting of brain multiple sclerosis lesion and spinal cord grey matter segmentation. On both datasets, results show a correlation (R2 = 0.60 and 0.93) between rater bias and deep learning uncertainty. The impact of label fusion between raters' annotations on this relationship is also explored, and we show that multi-center consensuses are more effective than single-center consensuses to reduce uncertainty, since rater style is mostly center-specific.