# Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstraction

arXiv: https://arxiv.org/abs/2001.00208 · HTML: https://mlsanity.com/paper/2001.00208/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jan 2020.

## Built on by (most cited first, 30 of 70)

- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer](https://mlsanity.com/paper/2301.11798/) (2023)
- [UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learner](https://mlsanity.com/paper/2304.03493/) (2023)
- [MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation](https://mlsanity.com/paper/2303.14444/) (2023)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model](https://mlsanity.com/paper/2404.09957/) (2024)
- [Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography](https://mlsanity.com/paper/2405.18356/) (2024)
- [Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans](https://mlsanity.com/paper/2302.00162/) (2023)
- [Label-Assemble: Leveraging Multiple Datasets with Partial Labels](https://mlsanity.com/paper/2109.12265/) (2021)
- [Learning Incrementally to Segment Multiple Organs in a CT Image](https://mlsanity.com/paper/2203.02100/) (2022)
- [Learning from partially labeled data for multi-organ and tumor segmentation](https://mlsanity.com/paper/2211.06894/) (2022)
- [Omni-Seg: A Scale-aware Dynamic Network for Renal Pathological Image Segmentation](https://mlsanity.com/paper/2206.13632/) (2022)
- [COSST: Multi-organ Segmentation with Partially Labeled Datasets Using Comprehensive Supervisions and Self-training](https://mlsanity.com/paper/2304.14030/) (2023)
- [Federated Multi-organ Segmentation with Inconsistent Labels](https://mlsanity.com/paper/2206.07156/) (2022)
- [ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Prompting](https://mlsanity.com/paper/2312.04964/) (2023)
- [PrPSeg: Universal Proposition Learning for Panoramic Renal Pathology Segmentation](https://mlsanity.com/paper/2402.19286/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [Label-set Loss Functions for Partial Supervision: Application to Fetal Brain 3D MRI Parcellation](https://mlsanity.com/paper/2107.03846/) (2021)
- [Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation](https://mlsanity.com/paper/2311.10696/) (2023)
- [Universal Segmentation of 33 Anatomies](https://mlsanity.com/paper/2203.02098/) (2022)
- [GAMED-Snake: Gradient-aware Adaptive Momentum Evolution Deep Snake Model for Multi-organ Segmentation](https://mlsanity.com/paper/2501.12844/) (2025)
- [Incremental Learning for Multi-organ Segmentation with Partially Labeled Datasets](https://mlsanity.com/paper/2103.04526/) (2021)
- [MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model](https://mlsanity.com/paper/2410.05905/) (2024)
- [ConDistFL: Conditional Distillation for Federated Learning from Partially Annotated Data](https://mlsanity.com/paper/2308.04070/) (2023)
- [Revisiting Vicinal Risk Minimization for Partially Supervised Multi-Label Classification Under Data Scarcity](https://mlsanity.com/paper/2204.08954/) (2022)
- [A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation](https://mlsanity.com/paper/2408.11210/) (2024)

## Abstract (arXiv)

Shortage of fully annotated datasets has been a limiting factor in developing deep learning based image segmentation algorithms and the problem becomes more pronounced in multi-organ segmentation. In this paper, we propose a unified training strategy that enables a novel multi-scale deep neural network to be trained on multiple partially labeled datasets for multi-organ segmentation. In addition, a new network architecture for multi-scale feature abstraction is proposed to integrate pyramid input and feature analysis into a U-shape pyramid structure. To bridge the semantic gap caused by directly merging features from different scales, an equal convolutional depth mechanism is introduced. Furthermore, we employ a deep supervision mechanism to refine the outputs in different scales. To fully leverage the segmentation features from all the scales, we design an adaptive weighting layer to fuse the outputs in an automatic fashion. All these mechanisms together are integrated into a Pyramid Input Pyramid Output Feature Abstraction Network (PIPO-FAN). Our proposed method was evaluated on four publicly available datasets, including BTCV, LiTS, KiTS and Spleen, where very promising performance has been achieved. The source code of this work is publicly shared at https://github.com/DIAL-RPI/PIPO-FAN for others to easily reproduce the work and build their own models with the introduced mechanisms.