# Diversified and Personalized Multi-rater Medical Image Segmentation

arXiv: https://arxiv.org/abs/2403.13417 · HTML: https://mlsanity.com/paper/2403.13417/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [PHiSeg: Capturing Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/1906.04045/) (2019)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [The Cramer Distance as a Solution to Biased Wasserstein Gradients](https://mlsanity.com/paper/1705.10743/) (2017)
- [Mutual Consistency Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2109.09960/) (2021)
- [Ambiguous Medical Image Segmentation using Diffusion Models](https://mlsanity.com/paper/2304.04745/) (2023)
- [A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities](https://mlsanity.com/paper/1905.13077/) (2019)
- [Disentangling Human Error from the Ground Truth in Segmentation of Medical Images](https://mlsanity.com/paper/2007.15963/) (2020)
- [CryoNuSeg: A Dataset for Nuclei Instance Segmentation of Cryosectioned H&E-Stained Histological Images](https://mlsanity.com/paper/2101.00442/) (2021)
- [Instance-Dependent Label-Noise Learning with Manifold-Regularized Transition Matrix Estimation](https://mlsanity.com/paper/2206.02791/) (2022)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalization](https://mlsanity.com/paper/2208.11351/) (2022)
- [Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds](https://mlsanity.com/paper/1905.13436/) (2019)
- [Adaptive Sample Selection for Robust Learning under Label Noise](https://mlsanity.com/paper/2106.15292/) (2021)
- [A Survey of Crowdsourcing in Medical Image Analysis](https://mlsanity.com/paper/1902.09159/) (2019)
- [Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation](https://mlsanity.com/paper/2307.11397/) (2023)
- [CoactSeg: Learning from Heterogeneous Data for New Multiple Sclerosis Lesion Segmentation](https://mlsanity.com/paper/2307.04513/) (2023)
- [Transformer-based Annotation Bias-aware Medical Image Segmentation](https://mlsanity.com/paper/2306.01340/) (2023)

## Built on by (most cited first, 9 of 9)

- [Autoregressive Medical Image Segmentation via Next-Scale Mask Prediction](https://mlsanity.com/paper/2502.20784/) (2025)
- [SAM-aware Test-time Adaptation for Universal Medical Image Segmentation](https://mlsanity.com/paper/2506.05221/) (2025)
- [DiffOSeg: Omni Medical Image Segmentation via Multi-Expert Collaboration Diffusion Model](https://mlsanity.com/paper/2507.13087/) (2025)
- [Neovascularization Segmentation via a Multilateral Interaction-Enhanced Graph Convolutional Network](https://mlsanity.com/paper/2508.03197/) (2025)
- [ProSona: Prompt-Guided Personalization for Multi-Expert Medical Image Segmentation](https://mlsanity.com/paper/2511.08046/) (2025)
- [Probabilistic Modeling of Multi-rater Medical Image Segmentation for Diversity and Personalization](https://mlsanity.com/paper/2512.00748/) (2025)
- [Harmonized Feature Conditioning and Frequency-Prompt Personalization for Multi-Rater Medical Segmentation](https://mlsanity.com/paper/2605.08210/) (2026)
- [Attention-Based Prototype Calibration for Multi-Rater Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2606.16325/) (2026)
- [Attribute Token Arithmetic: Disentangled and Continuous Semantic Control for Visual Autoregressive Models](https://mlsanity.com/paper/2608.28082/) (2026)

## Abstract (arXiv)

Annotation ambiguity due to inherent data uncertainties such as blurred boundaries in medical scans and different observer expertise and preferences has become a major obstacle for training deep-learning based medical image segmentation models. To address it, the common practice is to gather multiple annotations from different experts, leading to the setting of multi-rater medical image segmentation. Existing works aim to either merge different annotations into the "groundtruth" that is often unattainable in numerous medical contexts, or generate diverse results, or produce personalized results corresponding to individual expert raters. Here, we bring up a more ambitious goal for multi-rater medical image segmentation, i.e., obtaining both diversified and personalized results. Specifically, we propose a two-stage framework named D-Persona (first Diversification and then Personalization). In Stage I, we exploit multiple given annotations to train a Probabilistic U-Net model, with a bound-constrained loss to improve the prediction diversity. In this way, a common latent space is constructed in Stage I, where different latent codes denote diversified expert opinions. Then, in Stage II, we design multiple attention-based projection heads to adaptively query the corresponding expert prompts from the shared latent space, and then perform the personalized medical image segmentation. We evaluated the proposed model on our in-house Nasopharyngeal Carcinoma dataset and the public lung nodule dataset (i.e., LIDC-IDRI). Extensive experiments demonstrated our D-Persona can provide diversified and personalized results at the same time, achieving new SOTA performance for multi-rater medical image segmentation. Our code will be released at https://github.com/ycwu1997/D-Persona.