# Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle

arXiv: https://arxiv.org/abs/2208.03016 · HTML: https://mlsanity.com/paper/2208.03016/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 82 references that are in the map.
- First posted on arXiv 5 Aug 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [CE-Net: Context Encoder Network for 2D Medical Image Segmentation](https://mlsanity.com/paper/1903.02740/) (2019)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [PHiSeg: Capturing Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/1906.04045/) (2019)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data](https://mlsanity.com/paper/2002.03366/) (2020)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Who Said What: Modeling Individual Labelers Improves Classification](https://mlsanity.com/paper/1703.08774/) (2017)
- [Attention Based Glaucoma Detection: A Large-scale Database and CNN Model](https://mlsanity.com/paper/1903.10831/) (2019)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [Boundary and Entropy-driven Adversarial Learning for Fundus Image Segmentation](https://mlsanity.com/paper/1906.11143/) (2019)
- [Patch-based Output Space Adversarial Learning for Joint Optic Disc and Cup Segmentation](https://mlsanity.com/paper/1902.07519/) (2019)
- [Disc-aware Ensemble Network for Glaucoma Screening from Fundus Image](https://mlsanity.com/paper/1805.07549/) (2018)
- [REFUGE2 Challenge: A Treasure Trove for Multi-Dimension Analysis and Evaluation in Glaucoma Screening](https://mlsanity.com/paper/2202.08994/) (2022)
- [Attention Guided Network for Retinal Image Segmentation](https://mlsanity.com/paper/1907.12930/) (2019)
- [Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds](https://mlsanity.com/paper/1905.13436/) (2019)
- [Star Shape Prior in Fully Convolutional Networks for Skin Lesion Segmentation](https://mlsanity.com/paper/1806.08437/) (2018)
- [Better Than Reference In Low Light Image Enhancement: Conditional Re-Enhancement Networks](https://mlsanity.com/paper/2008.11434/) (2020)
- [Difficulty-aware Glaucoma Classification with Multi-Rater Consensus Modeling](https://mlsanity.com/paper/2007.14848/) (2020)

## Built on by (most cited first, 3 of 3)

- [Harmonized Feature Conditioning and Frequency-Prompt Personalization for Multi-Rater Medical Segmentation](https://mlsanity.com/paper/2605.08210/) (2026)
- [Structure-aware Knowledge-guided Heterogeneous Mamba for Zygomaticomaxillary Suture Assessment](https://mlsanity.com/paper/2606.16749/) (2026)
- [Observer Choice and Threshold Selection in Retinal Vessel Segmentation: A Subject-Separated Evaluation](https://mlsanity.com/paper/2609.25597/) (2026)

## Abstract (arXiv)

On the medical images, many of the tissues/lesions may be ambiguous. That is why the medical segmentation is typically annotated by a group of clinical experts to mitigate the personal bias. However, this clinical routine also brings new challenges to the application of machine learning algorithms. Without a definite ground-truth, it will be difficult to train and evaluate the deep learning models. When the annotations are collected from different graders, a common choice is majority vote. However such a strategy ignores the difference between the grader expertness. In this paper, we consider the task of predicting the segmentation with the calibrated inter-observer uncertainty. We note that in clinical practice, the medical image segmentation is usually used to assist the disease diagnosis. Inspired by this observation, we propose diagnosis-first principle, which is to take disease diagnosis as the criterion to calibrate the inter-observer segmentation uncertainty. Following this idea, a framework named Diagnosis First segmentation Framework (DiFF) is proposed to estimate diagnosis-first segmentation from the raw images.Specifically, DiFF will first learn to fuse the multi-rater segmentation labels to a single ground-truth which could maximize the disease diagnosis performance. We dubbed the fused ground-truth as Diagnosis First Ground-truth (DF-GT).Then, we further propose Take and Give Modelto segment DF-GT from the raw image. We verify the effectiveness of DiFF on three different medical segmentation tasks: OD/OC segmentation on fundus images, thyroid nodule segmentation on ultrasound images, and skin lesion segmentation on dermoscopic images. Experimental results show that the proposed DiFF is able to significantly facilitate the corresponding disease diagnosis, which outperforms previous state-of-the-art multi-rater learning methods.