# Learning Incrementally to Segment Multiple Organs in a CT Image

arXiv: https://arxiv.org/abs/2203.02100 · HTML: https://mlsanity.com/paper/2203.02100/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises](https://mlsanity.com/paper/2008.09104/) (2020)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Reinforced Continual Learning](https://mlsanity.com/paper/1805.12369/) (2018)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentation](https://mlsanity.com/paper/1904.06346/) (2019)
- [Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstraction](https://mlsanity.com/paper/2001.00208/) (2020)
- [Incremental learning for the detection and classification of GAN-generated images](https://mlsanity.com/paper/1910.01568/) (2019)
- [Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data](https://mlsanity.com/paper/1810.12448/) (2018)
- [Extending Pretrained Segmentation Networks with Additional Anatomical Structures](https://mlsanity.com/paper/1811.04634/) (2018)

## Built on by (most cited first, 19 of 19)

- [Continual Learning for Abdominal Multi-Organ and Tumor Segmentation](https://mlsanity.com/paper/2306.00988/) (2023)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography](https://mlsanity.com/paper/2405.18356/) (2024)
- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans](https://mlsanity.com/paper/2302.00162/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Low-Rank Mixture-of-Experts for Continual Medical Image Segmentation](https://mlsanity.com/paper/2406.13583/) (2024)
- [Incremental Learning for Heterogeneous Structure Segmentation in Brain Tumor MRI](https://mlsanity.com/paper/2305.19404/) (2023)
- [Leveraging AI Predicted and Expert Revised Annotations in Interactive Segmentation: Continual Tuning or Full Training?](https://mlsanity.com/paper/2402.19423/) (2024)
- [Continual Learning in Medical Imaging: A Survey and Practical Analysis](https://mlsanity.com/paper/2405.13482/) (2024)
- [Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting](https://mlsanity.com/paper/2406.19796/) (2024)
- [Dynamic Robot-Assisted Surgery with Hierarchical Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2508.01713/) (2025)
- [Multi-site Organ Segmentation with Federated Partial Supervision and Site Adaptation](https://mlsanity.com/paper/2302.03911/) (2023)
- [Low-Rank Continual Pyramid Vision Transformer: Incrementally Segment Whole-Body Organs in CT with Light-Weighted Adaptation](https://mlsanity.com/paper/2410.04689/) (2024)
- [Unlocking Positive Transfer in Incrementally Learning Surgical Instruments: A Self-reflection Hierarchical Prompt Framework](https://mlsanity.com/paper/2604.02877/) (2026)
- [Federated Learning with Partially Labeled Data: A Conditional Distillation Approach](https://mlsanity.com/paper/2412.18833/) (2024)
- [Dynamic Allocation Hypernetwork with Adaptive Model Recalibration for FCL](https://mlsanity.com/paper/2503.18064/) (2025)
- [Beyond Forgetting in Continual Medical Image Segmentation: A Comprehensive Benchmark Study](https://mlsanity.com/paper/2605.06160/) (2026)
- [C^2GR: Coupled Comprehensive Generative Replay for a Continually Learnable Universal Segmentation Model](https://mlsanity.com/paper/2606.23473/) (2026)

## Abstract (arXiv)

There exists a large number of datasets for organ segmentation, which are partially annotated and sequentially constructed. A typical dataset is constructed at a certain time by curating medical images and annotating the organs of interest. In other words, new datasets with annotations of new organ categories are built over time. To unleash the potential behind these partially labeled, sequentially-constructed datasets, we propose to incrementally learn a multi-organ segmentation model. In each incremental learning (IL) stage, we lose the access to previous data and annotations, whose knowledge is assumingly captured by the current model, and gain the access to a new dataset with annotations of new organ categories, from which we learn to update the organ segmentation model to include the new organs. While IL is notorious for its `catastrophic forgetting' weakness in the context of natural image analysis, we experimentally discover that such a weakness mostly disappears for CT multi-organ segmentation. To further stabilize the model performance across the IL stages, we introduce a light memory module and some loss functions to restrain the representation of different categories in feature space, aggregating feature representation of the same class and separating feature representation of different classes. Extensive experiments on five open-sourced datasets are conducted to illustrate the effectiveness of our method.