# Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation

arXiv: https://arxiv.org/abs/2311.10696 · HTML: https://mlsanity.com/paper/2311.10696/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation](https://mlsanity.com/paper/1912.05074/) (2019)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation](https://mlsanity.com/paper/2303.09975/) (2023)
- [DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation](https://mlsanity.com/paper/1506.06448/) (2015)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentation](https://mlsanity.com/paper/1904.06346/) (2019)
- [Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstraction](https://mlsanity.com/paper/2001.00208/) (2020)
- [Scribble-Supervised Medical Image Segmentation via Dual-Branch Network and Dynamically Mixed Pseudo Labels Supervision](https://mlsanity.com/paper/2203.02106/) (2022)
- [Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge](https://mlsanity.com/paper/2308.05862/) (2023)
- [Bounding boxes for weakly supervised segmentation: Global constraints get close to full supervision](https://mlsanity.com/paper/2004.06816/) (2020)
- [A deep learning model integrating FCNNs and CRFs for brain tumor segmentation](https://mlsanity.com/paper/1702.04528/) (2017)
- [Multi-organ Segmentation via Co-training Weight-averaged Models from Few-organ Datasets](https://mlsanity.com/paper/2008.07149/) (2020)
- [Fine-Grained Segmentation Networks: Self-Supervised Segmentation for Improved Long-Term Visual Localization](https://mlsanity.com/paper/1908.06387/) (2019)
- [Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans](https://mlsanity.com/paper/2302.00162/) (2023)
- [Weakly supervised segmentation with point annotations for histopathology images via contrast-based variational model](https://mlsanity.com/paper/2304.03572/) (2023)
- [ACEnet: Anatomical Context-Encoding Network for Neuroanatomy Segmentation](https://mlsanity.com/paper/2002.05773/) (2020)
- [SkullEngine: A Multi-stage CNN Framework for Collaborative CBCT Image Segmentation and Landmark Detection](https://mlsanity.com/paper/2110.03828/) (2021)
- [Annotation by Clicks: A Point-Supervised Contrastive Variance Method for Medical Semantic Segmentation](https://mlsanity.com/paper/2212.08774/) (2022)
- [Brain Tissue Segmentation Across the Human Lifespan via Supervised Contrastive Learning](https://mlsanity.com/paper/2301.01369/) (2023)

## Built on by (most cited first, 12 of 12)

- [Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography](https://mlsanity.com/paper/2405.18356/) (2024)
- [MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model](https://mlsanity.com/paper/2410.05905/) (2024)
- [Aggregated Attributions for Explanatory Analysis of 3D Segmentation Models](https://mlsanity.com/paper/2407.16653/) (2024)
- [Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imaging](https://mlsanity.com/paper/2512.13855/) (2025)
- [A Classifier-Free Incremental Learning Framework for Scalable Medical Image Segmentation](https://mlsanity.com/paper/2405.16328/) (2024)
- [Multi-Cohort Framework with Cohort-Aware Attention and Adversarial Mutual-Information Minimization for Whole Slide Image Classification](https://mlsanity.com/paper/2409.11119/) (2024)
- [Binarization-Aware Adjuster for Discrete Decision Learning with an Application to Edge Detection](https://mlsanity.com/paper/2506.12460/) (2025)
- [COME: Dual Structure-Semantic Learning with Collaborative MoE for Universal Lesion Detection Across Heterogeneous Ultrasound Datasets](https://mlsanity.com/paper/2508.09886/) (2025)
- [RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation](https://mlsanity.com/paper/2604.09814/) (2026)
- [DiSC-Med: Diffusion-based Semantic Communications for Robust Medical Image Transmission](https://mlsanity.com/paper/2508.00172/) (2025)
- [Exploiting Unlabeled Structures through Task Consistency Training for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2509.04732/) (2025)
- [SemiTooth: a Generalizable Semi-supervised Framework for Multi-Source Tooth Segmentation](https://mlsanity.com/paper/2603.11616/) (2026)

## Abstract (arXiv)

A versatile medical image segmentation model applicable to images acquired with diverse equipment and protocols can facilitate model deployment and maintenance. However, building such a model typically demands a large, diverse, and fully annotated dataset, which is challenging to obtain due to the labor-intensive nature of data curation. To address this challenge, we propose a cost-effective alternative that harnesses multi-source data with only partial or sparse segmentation labels for training, substantially reducing the cost of developing a versatile model. We devise strategies for model self-disambiguation, prior knowledge incorporation, and imbalance mitigation to tackle challenges associated with inconsistently labeled multi-source data, including label ambiguity and modality, dataset, and class imbalances. Experimental results on a multi-modal dataset compiled from eight different sources for abdominal structure segmentation have demonstrated the effectiveness and superior performance of our method compared to state-of-the-art alternative approaches. We anticipate that its cost-saving features, which optimize the utilization of existing annotated data and reduce annotation efforts for new data, will have a significant impact in the field.