# What is Wrong with Continual Learning in Medical Image Segmentation?

arXiv: https://arxiv.org/abs/2010.11008 · HTML: https://mlsanity.com/paper/2010.11008/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2020.

## Built on by (most cited first, 25 of 25)

- [LifeLonger: A Benchmark for Continual Disease Classification](https://mlsanity.com/paper/2204.05737/) (2022)
- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay](https://mlsanity.com/paper/2308.14100/) (2023)
- [Domain-incremental Cardiac Image Segmentation with Style-oriented Replay and Domain-sensitive Feature Whitening](https://mlsanity.com/paper/2211.04862/) (2022)
- [Adversarial Continual Learning for Multi-Domain Hippocampal Segmentation](https://mlsanity.com/paper/2107.08751/) (2021)
- [Continual Hippocampus Segmentation with Transformers](https://mlsanity.com/paper/2204.08043/) (2022)
- [Continual Active Learning for Efficient Adaptation of Machine Learning Models to Changing Image Acquisition](https://mlsanity.com/paper/2106.03351/) (2021)
- [Continual Learning in Medical Imaging: A Survey and Practical Analysis](https://mlsanity.com/paper/2405.13482/) (2024)
- [Generative appearance replay for continual unsupervised domain adaptation](https://mlsanity.com/paper/2301.01211/) (2023)
- [Controllable Continual Test-Time Adaptation](https://mlsanity.com/paper/2405.14602/) (2024)
- [Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments](https://mlsanity.com/paper/2110.11006/) (2021)
- [Is Continual Learning Ready for Real-world Challenges?](https://mlsanity.com/paper/2402.10130/) (2024)
- [Continual Active Learning Using Pseudo-Domains for Limited Labelling Resources and Changing Acquisition Characteristics](https://mlsanity.com/paper/2111.13069/) (2021)
- [Memory-Efficient Prompt Tuning for Incremental Histopathology Classification](https://mlsanity.com/paper/2401.11674/) (2024)
- [Distribution-Aware Replay for Continual MRI Segmentation](https://mlsanity.com/paper/2407.21216/) (2024)
- [Fairness Evolution in Continual Learning for Medical Imaging](https://mlsanity.com/paper/2406.02480/) (2024)
- [NCAdapt: Dynamic adaptation with domain-specific Neural Cellular Automata for continual hippocampus segmentation](https://mlsanity.com/paper/2410.23368/) (2024)
- [Decoupling Continual Semantic Segmentation](https://mlsanity.com/paper/2508.05065/) (2025)
- [TADIL: Task-Agnostic Domain-Incremental Learning through Task-ID Inference using Transformer Nearest-Centroid Embeddings](https://mlsanity.com/paper/2306.11955/) (2023)
- [A dynamic interactive learning framework for automated 3D medical image segmentation](https://mlsanity.com/paper/2312.06072/) (2023)
- [Exploring UMAP in hybrid models of entropy-based and representativeness sampling for active learning in biomedical segmentation](https://mlsanity.com/paper/2312.10361/) (2023)
- [From Lab to Pocket: A Novel Continual Learning-based Mobile Application for Screening COVID-19](https://mlsanity.com/paper/2410.12589/) (2024)
- [Beyond Forgetting in Continual Medical Image Segmentation: A Comprehensive Benchmark Study](https://mlsanity.com/paper/2605.06160/) (2026)
- [What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentation](https://mlsanity.com/paper/2608.13660/) (2026)

## Abstract (arXiv)

Continual learning protocols are attracting increasing attention from the medical imaging community. In continual environments, datasets acquired under different conditions arrive sequentially; and each is only available for a limited period of time. Given the inherent privacy risks associated with medical data, this setup reflects the reality of deployment for deep learning diagnostic radiology systems. Many techniques exist to learn continuously for image classification, and several have been adapted to semantic segmentation. Yet most struggle to accumulate knowledge in a meaningful manner. Instead, they focus on preventing the problem of catastrophic forgetting, even when this reduces model plasticity and thereon burdens the training process. This puts into question whether the additional overhead of knowledge preservation is worth it - particularly for medical image segmentation, where computation requirements are already high - or if maintaining separate models would be a better solution. We propose UNEG, a simple and widely applicable multi-model benchmark that maintains separate segmentation and autoencoder networks for each training stage. The autoencoder is built from the same architecture as the segmentation network, which in our case is a full-resolution nnU-Net, to bypass any additional design decisions. During inference, the reconstruction error is used to select the most appropriate segmenter for each test image. Open this concept, we develop a fair evaluation scheme for different continual learning settings that moves beyond the prevention of catastrophic forgetting. Our results across three regions of interest (prostate, hippocampus, and right ventricle) show that UNEG outperforms several continual learning methods, reinforcing the need for strong baselines in continual learning research.