# That Label's Got Style: Handling Label Style Bias for Uncertain Image Segmentation

arXiv: https://arxiv.org/abs/2303.15850 · HTML: https://mlsanity.com/paper/2303.15850/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 21 of 21)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference](https://mlsanity.com/paper/1506.02158/) (2015)
- [PHiSeg: Capturing Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/1906.04045/) (2019)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses](https://mlsanity.com/paper/1612.00197/) (2016)
- [Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles](https://mlsanity.com/paper/1606.07839/) (2016)
- [A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities](https://mlsanity.com/paper/1905.13077/) (2019)
- [Is segmentation uncertainty useful?](https://mlsanity.com/paper/2103.16265/) (2021)
- [Is MC Dropout Bayesian?](https://mlsanity.com/paper/2110.04286/) (2021)
- [Uncertainty quantification in medical image segmentation with normalizing flows](https://mlsanity.com/paper/2006.02683/) (2020)
- [Dropout as a Bayesian Approximation: Appendix](https://mlsanity.com/paper/1506.02157/) (2015)
- [Distributional Gaussian Process Layers for Outlier Detection in Image Segmentation](https://mlsanity.com/paper/2104.13756/) (2021)
- [Joint M-Best-Diverse Labelings as a Parametric Submodular Minimization](https://mlsanity.com/paper/1606.07015/) (2016)

## Built on by (most cited first, 13 of 13)

- [A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation](https://mlsanity.com/paper/2411.16370/) (2024)
- [Flow Stochastic Segmentation Networks](https://mlsanity.com/paper/2507.18838/) (2025)
- [Segmentation Style Discovery: Application to Skin Lesion Images](https://mlsanity.com/paper/2408.02787/) (2024)
- [What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?](https://mlsanity.com/paper/2508.09381/) (2025)
- [Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation](https://mlsanity.com/paper/2605.06891/) (2026)
- [Navigating Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/2407.16367/) (2024)
- [Ambiguity-aware Truncated Flow Matching for Ambiguous Medical Image Segmentation](https://mlsanity.com/paper/2511.06857/) (2025)
- [Multi-style conversion for semantic segmentation of lesions in fundus images by adversarial attacks](https://mlsanity.com/paper/2410.13822/) (2024)
- [PCA-Enhanced Probabilistic U-Net for Effective Ambiguous Medical Image Segmentation](https://mlsanity.com/paper/2603.11550/) (2026)
- [Harmonized Feature Conditioning and Frequency-Prompt Personalization for Multi-Rater Medical Segmentation](https://mlsanity.com/paper/2605.08210/) (2026)
- [Interpretable Probabilistic Medical Image Segmentation via Gaussian Process with Explicit Modelling of Annotation Bias and Variability](https://mlsanity.com/paper/2606.23177/) (2026)
- [KANResDiff: Learning Local Residual Diffusion via Kolmogorov-Arnold Network for Ambiguous Medical Image Segmentation](https://mlsanity.com/paper/2608.11617/) (2026)
- [A Multi-Annotator Study of Segmentation Noise and Uncertainty in Turbid Underwater Images](https://mlsanity.com/paper/2608.15363/) (2026)

## Abstract (arXiv)

Segmentation uncertainty models predict a distribution over plausible segmentations for a given input, which they learn from the annotator variation in the training set. However, in practice these annotations can differ systematically in the way they are generated, for example through the use of different labeling tools. This results in datasets that contain both data variability and differing label styles. In this paper, we demonstrate that applying state-of-the-art segmentation uncertainty models on such datasets can lead to model bias caused by the different label styles. We present an updated modelling objective conditioning on labeling style for aleatoric uncertainty estimation, and modify two state-of-the-art-architectures for segmentation uncertainty accordingly. We show with extensive experiments that this method reduces label style bias, while improving segmentation performance, increasing the applicability of segmentation uncertainty models in the wild. We curate two datasets, with annotations in different label styles, which we will make publicly available along with our code upon publication.