# Learning from partially labeled data for multi-organ and tumor segmentation

arXiv: https://arxiv.org/abs/2211.06894 · HTML: https://mlsanity.com/paper/2211.06894/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 81 references that are in the map.
- First posted on arXiv 13 Nov 2022.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)

## Built on by (most cited first, 18 of 18)

- [Large-Scale 3D Medical Image Pre-training with Geometric Context Priors](https://mlsanity.com/paper/2410.09890/) (2024)
- [Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography](https://mlsanity.com/paper/2405.18356/) (2024)
- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [Automatic Organ and Pan-cancer Segmentation in Abdomen CT: the FLARE 2023 Challenge](https://mlsanity.com/paper/2408.12534/) (2024)
- [Show and Segment: Universal Medical Image Segmentation via In-Context Learning](https://mlsanity.com/paper/2503.19359/) (2025)
- [MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model](https://mlsanity.com/paper/2410.05905/) (2024)
- [SRSNetwork: Siamese Reconstruction-Segmentation Networks based on Dynamic-Parameter Convolution](https://mlsanity.com/paper/2312.01741/) (2023)
- [Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval](https://mlsanity.com/paper/2409.09430/) (2024)
- [Towards more precise automatic analysis: a comprehensive survey of deep learning-based multi-organ segmentation](https://mlsanity.com/paper/2303.00232/) (2023)
- [UniUSNet: A Promptable Framework for Universal Ultrasound Disease Prediction and Tissue Segmentation](https://mlsanity.com/paper/2406.01154/) (2024)
- [MOSMOS: Multi-organ segmentation facilitated by medical report supervision](https://mlsanity.com/paper/2409.02418/) (2024)
- [Federated Learning with Partially Labeled Data: A Conditional Distillation Approach](https://mlsanity.com/paper/2412.18833/) (2024)
- [Modality-Projection Universal Model for Comprehensive Full-Body Medical Imaging Segmentation](https://mlsanity.com/paper/2412.19026/) (2024)
- [MO-CTranS: A unified multi-organ segmentation model learning from multiple heterogeneously labelled datasets](https://mlsanity.com/paper/2503.22557/) (2025)
- [CDPDNet: Integrating Text Guidance with Hybrid Vision Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2505.18958/) (2025)
- [DpDNet: An Dual-Prompt-Driven Network for Universal PET-CT Segmentation](https://mlsanity.com/paper/2507.07126/) (2025)
- [Unified Start, Personalized End: Progressive Pruning for Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2509.09267/) (2025)
- [Foundation Model-guided Iteratively Prompting and Pseudo-Labeling for Partially Labeled Medical Image Segmentation](https://mlsanity.com/paper/2604.01038/) (2026)

## Abstract (arXiv)

Medical image benchmarks for the segmentation of organs and tumors suffer from the partially labeling issue due to its intensive cost of labor and expertise. Current mainstream approaches follow the practice of one network solving one task. With this pipeline, not only the performance is limited by the typically small dataset of a single task, but also the computation cost linearly increases with the number of tasks. To address this, we propose a Transformer based dynamic on-demand network (TransDoDNet) that learns to segment organs and tumors on multiple partially labeled datasets. Specifically, TransDoDNet has a hybrid backbone that is composed of the convolutional neural network and Transformer. A dynamic head enables the network to accomplish multiple segmentation tasks flexibly. Unlike existing approaches that fix kernels after training, the kernels in the dynamic head are generated adaptively by the Transformer, which employs the self-attention mechanism to model long-range organ-wise dependencies and decodes the organ embedding that can represent each organ. We create a large-scale partially labeled Multi-Organ and Tumor Segmentation benchmark, termed MOTS, and demonstrate the superior performance of our TransDoDNet over other competitors on seven organ and tumor segmentation tasks. This study also provides a general 3D medical image segmentation model, which has been pre-trained on the large-scale MOTS benchmark and has demonstrated advanced performance over BYOL, the current predominant self-supervised learning method. Code will be available at urlhttps://git.io/DoDNet.