# Universal Segmentation of 33 Anatomies

arXiv: https://arxiv.org/abs/2203.02098 · HTML: https://mlsanity.com/paper/2203.02098/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises](https://mlsanity.com/paper/2008.09104/) (2020)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images](https://mlsanity.com/paper/2001.09193/) (2020)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentation](https://mlsanity.com/paper/1904.06346/) (2019)
- [Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstraction](https://mlsanity.com/paper/2001.00208/) (2020)
- [Deep Learning to Segment Pelvic Bones: Large-scale CT Datasets and Baseline Models](https://mlsanity.com/paper/2012.08721/) (2020)
- [3D Segmentation Networks for Excessive Numbers of Classes: Distinct Bone Segmentation in Upper Bodies](https://mlsanity.com/paper/2010.07045/) (2020)

## Built on by (most cited first, 10 of 10)

- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learner](https://mlsanity.com/paper/2304.03493/) (2023)
- [M4oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Experts](https://mlsanity.com/paper/2405.09446/) (2024)
- [COSST: Multi-organ Segmentation with Partially Labeled Datasets Using Comprehensive Supervisions and Self-training](https://mlsanity.com/paper/2304.14030/) (2023)
- [MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model](https://mlsanity.com/paper/2410.05905/) (2024)
- [Med-Query: Steerable Parsing of 9-DoF Medical Anatomies with Query Embedding](https://mlsanity.com/paper/2212.02014/) (2022)
- [Multi-site Organ Segmentation with Federated Partial Supervision and Site Adaptation](https://mlsanity.com/paper/2302.03911/) (2023)
- [Deep Mutual Learning among Partially Labeled Datasets for Multi-Organ Segmentation](https://mlsanity.com/paper/2407.12611/) (2024)
- [Exploiting Unlabeled Structures through Task Consistency Training for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2509.04732/) (2025)
- [Optimal Transport Metric Learning for Feature Alignment in Partially Supervised Segmentation](https://mlsanity.com/paper/2609.19176/) (2026)

## Abstract (arXiv)

In the paper, we present an approach for learning a single model that universally segments 33 anatomical structures, including vertebrae, pelvic bones, and abdominal organs. Our model building has to address the following challenges. Firstly, while it is ideal to learn such a model from a large-scale, fully-annotated dataset, it is practically hard to curate such a dataset. Thus, we resort to learn from a union of multiple datasets, with each dataset containing the images that are partially labeled. Secondly, along the line of partial labelling, we contribute an open-source, large-scale vertebra segmentation dataset for the benefit of spine analysis community, CTSpine1K, boasting over 1,000 3D volumes and over 11K annotated vertebrae. Thirdly, in a 3D medical image segmentation task, due to the limitation of GPU memory, we always train a model using cropped patches as inputs instead a whole 3D volume, which limits the amount of contextual information to be learned. To this, we propose a cross-patch transformer module to fuse more information in adjacent patches, which enlarges the aggregated receptive field for improved segmentation performance. This is especially important for segmenting, say, the elongated spine. Based on 7 partially labeled datasets that collectively contain about 2,800 3D volumes, we successfully learn such a universal model. Finally, we evaluate the universal model on multiple open-source datasets, proving that our model has a good generalization performance and can potentially serve as a solid foundation for downstream tasks.