# Transformer-based Annotation Bias-aware Medical Image Segmentation

arXiv: https://arxiv.org/abs/2306.01340 · HTML: https://mlsanity.com/paper/2306.01340/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [Deep learning with noisy labels: exploring techniques and remedies in medical image analysis](https://mlsanity.com/paper/1912.02911/) (2019)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [Annotation-efficient deep learning for automatic medical image segmentation](https://mlsanity.com/paper/2012.04885/) (2020)
- [Who Said What: Modeling Individual Labelers Improves Classification](https://mlsanity.com/paper/1703.08774/) (2017)
- [Disentangling Human Error from the Ground Truth in Segmentation of Medical Images](https://mlsanity.com/paper/2007.15963/) (2020)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [Learning to Segment Skin Lesions from Noisy Annotations](https://mlsanity.com/paper/1906.03815/) (2019)
- [Pathological Image Segmentation with Noisy Labels](https://mlsanity.com/paper/2104.02602/) (2021)

## Built on by (most cited first, 8 of 8)

- [Diversified and Personalized Multi-rater Medical Image Segmentation](https://mlsanity.com/paper/2403.13417/) (2024)
- [Towards Clinician-Preferred Segmentation: Leveraging Human-in-the-Loop for Test Time Adaptation in Medical Image Segmentation](https://mlsanity.com/paper/2405.08270/) (2024)
- [URL: Combating Label Noise for Lung Nodule Malignancy Grading](https://mlsanity.com/paper/2308.08772/) (2023)
- [EDUE: Expert Disagreement-Guided One-Pass Uncertainty Estimation for Medical Image Segmentation](https://mlsanity.com/paper/2403.16594/) (2024)
- [DiffOSeg: Omni Medical Image Segmentation via Multi-Expert Collaboration Diffusion Model](https://mlsanity.com/paper/2507.13087/) (2025)
- [ProSona: Prompt-Guided Personalization for Multi-Expert Medical Image Segmentation](https://mlsanity.com/paper/2511.08046/) (2025)
- [Probabilistic Modeling of Multi-rater Medical Image Segmentation for Diversity and Personalization](https://mlsanity.com/paper/2512.00748/) (2025)
- [Harmonized Feature Conditioning and Frequency-Prompt Personalization for Multi-Rater Medical Segmentation](https://mlsanity.com/paper/2605.08210/) (2026)

## Abstract (arXiv)

Manual medical image segmentation is subjective and suffers from annotator-related bias, which can be mimicked or amplified by deep learning methods. Recently, researchers have suggested that such bias is the combination of the annotator preference and stochastic error, which are modeled by convolution blocks located after decoder and pixel-wise independent Gaussian distribution, respectively. It is unlikely that convolution blocks can effectively model the varying degrees of preference at the full resolution level. Additionally, the independent pixel-wise Gaussian distribution disregards pixel correlations, leading to a discontinuous boundary. This paper proposes a Transformer-based Annotation Bias-aware (TAB) medical image segmentation model, which tackles the annotator-related bias via modeling annotator preference and stochastic errors. TAB employs the Transformer with learnable queries to extract the different preference-focused features. This enables TAB to produce segmentation with various preferences simultaneously using a single segmentation head. Moreover, TAB takes the multivariant normal distribution assumption that models pixel correlations, and learns the annotation distribution to disentangle the stochastic error. We evaluated our TAB on an OD/OC segmentation benchmark annotated by six annotators. Our results suggest that TAB outperforms existing medical image segmentation models which take into account the annotator-related bias.