# Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation

arXiv: https://arxiv.org/abs/2210.17398 · HTML: https://mlsanity.com/paper/2210.17398/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 53 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Deep learning with noisy labels: exploring techniques and remedies in medical image analysis](https://mlsanity.com/paper/1912.02911/) (2019)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Unsupervised domain adaptation in brain lesion segmentation with adversarial networks](https://mlsanity.com/paper/1612.08894/) (2016)
- [DIVA: Domain Invariant Variational Autoencoders](https://mlsanity.com/paper/1905.10427/) (2019)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Transfer Learning from Synthetic to Real-Noise Denoising with Adaptive Instance Normalization](https://mlsanity.com/paper/2002.11244/) (2020)
- [Exploiting Domain-Specific Features to Enhance Domain Generalization](https://mlsanity.com/paper/2110.09410/) (2021)
- [Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks](https://mlsanity.com/paper/1805.07925/) (2018)
- [On the limits of cross-domain generalization in automated X-ray prediction](https://mlsanity.com/paper/2002.02497/) (2020)
- [No New-Net](https://mlsanity.com/paper/1809.10483/) (2018)
- [Disentangling Human Error from the Ground Truth in Segmentation of Medical Images](https://mlsanity.com/paper/2007.15963/) (2020)
- [A Lifelong Learning Approach to Brain MR Segmentation Across Scanners and Protocols](https://mlsanity.com/paper/1805.10170/) (2018)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [Brain Tumor Segmentation on MRI with Missing Modalities](https://mlsanity.com/paper/1904.07290/) (2019)
- [Exploring large scale public medical image datasets](https://mlsanity.com/paper/1907.12720/) (2019)
- [ALADIN: All Layer Adaptive Instance Normalization for Fine-grained Style Similarity](https://mlsanity.com/paper/2103.09776/) (2021)
- [Embracing the Disharmony in Medical Imaging: A Simple and Effective Framework for Domain Adaptation](https://mlsanity.com/paper/2103.12857/) (2021)
- [Imperfect Segmentation Labels: How Much Do They Matter?](https://mlsanity.com/paper/1806.04618/) (2018)
- [Modeling Annotator Preference and Stochastic Annotation Error for Medical Image Segmentation](https://mlsanity.com/paper/2111.13410/) (2021)
- [Impact of individual rater style on deep learning uncertainty in medical imaging segmentation](https://mlsanity.com/paper/2105.02197/) (2021)
- [INSIDE: Steering Spatial Attention with Non-Imaging Information in CNNs](https://mlsanity.com/paper/2008.10418/) (2020)
- [Cohort Bias Adaptation in Aggregated Datasets for Lesion Segmentation](https://mlsanity.com/paper/2108.00713/) (2021)

## Built on by (most cited first, 10 of 10)

- [Tyche: Stochastic In-Context Learning for Medical Image Segmentation](https://mlsanity.com/paper/2401.13650/) (2024)
- [How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation](https://mlsanity.com/paper/2308.06964/) (2023)
- [Exploring the interplay of label bias with subgroup size and separability: A case study in mammographic density classification](https://mlsanity.com/paper/2507.17996/) (2025)
- [Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation](https://mlsanity.com/paper/2605.06891/) (2026)
- [Continuous and complete liver vessel segmentation with graph-attention guided diffusion](https://mlsanity.com/paper/2411.00617/) (2024)
- [Reliability of deep learning models for anatomical landmark detection: The role of inter-rater variability](https://mlsanity.com/paper/2411.17850/) (2024)
- [Tumor-anchored deep feature random forests for out-of-distribution detection in lung cancer segmentation](https://mlsanity.com/paper/2512.08216/) (2025)
- [Temporal Inversion for Learning Interval Change in Chest X-Rays](https://mlsanity.com/paper/2604.04563/) (2026)
- [Expert-Adaptive Medical Image Segmentation](https://mlsanity.com/paper/2402.07330/) (2024)
- [The 1st Solution for CARE Liver Task Challenge 2025: Contrast-Aware Semi-Supervised Segmentation with Domain Generalization and Test-Time Adaptation](https://mlsanity.com/paper/2510.04243/) (2025)

## Abstract (arXiv)

Generalization is an important attribute of machine learning models, particularly for those that are to be deployed in a medical context, where unreliable predictions can have real world consequences. While the failure of models to generalize across datasets is typically attributed to a mismatch in the data distributions, performance gaps are often a consequence of biases in the 'ground-truth' label annotations. This is particularly important in the context of medical image segmentation of pathological structures (e.g. lesions), where the annotation process is much more subjective, and affected by a number underlying factors, including the annotation protocol, rater education/experience, and clinical aims, among others. In this paper, we show that modeling annotation biases, rather than ignoring them, poses a promising way of accounting for differences in annotation style across datasets. To this end, we propose a generalized conditioning framework to (1) learn and account for different annotation styles across multiple datasets using a single model, (2) identify similar annotation styles across different datasets in order to permit their effective aggregation, and (3) fine-tune a fully trained model to a new annotation style with just a few samples. Next, we present an image-conditioning approach to model annotation styles that correlate with specific image features, potentially enabling detection biases to be more easily identified.