# What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentation

arXiv: https://arxiv.org/abs/2608.13660 · HTML: https://mlsanity.com/paper/2608.13660/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 22 of 22)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](https://mlsanity.com/paper/2404.09556/) (2024)
- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Are All Layers Created Equal?](https://mlsanity.com/paper/1902.01996/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/2507.10485/) (2025)
- [Layerwise Optimization by Gradient Decomposition for Continual Learning](https://mlsanity.com/paper/2105.07561/) (2021)
- [Architecture Matters in Continual Learning](https://mlsanity.com/paper/2202.00275/) (2022)
- [MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation](https://mlsanity.com/paper/2303.14444/) (2023)
- [What is Wrong with Continual Learning in Medical Image Segmentation?](https://mlsanity.com/paper/2010.11008/) (2020)
- [How to Fine-Tune Vision Models with SGD](https://mlsanity.com/paper/2211.09359/) (2022)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Does Continual Learning Equally Forget All Parameters?](https://mlsanity.com/paper/2304.04158/) (2023)
- [Continual Hippocampus Segmentation with Transformers](https://mlsanity.com/paper/2204.08043/) (2022)
- [Distribution-Aware Replay for Continual MRI Segmentation](https://mlsanity.com/paper/2407.21216/) (2024)
- [Lean Unet: A Compact Model for Image Segmentation](https://mlsanity.com/paper/2512.03834/) (2025)
- [ECPC-IDS:A benchmark endometrail cancer PET/CT image dataset for evaluation of semantic segmentation and detection of hypermetabolic regions](https://mlsanity.com/paper/2308.08313/) (2023)
- [MBA-Net: SAM-driven Bidirectional Aggregation Network for Ovarian Tumor Segmentation](https://mlsanity.com/paper/2407.05984/) (2024)

## Abstract (arXiv)

The clinical management of gynecological diseases often relies on medical imaging for diagnosis, treatment planning, and follow-up. Segmentation in this setting is challenging because successive tasks may differ in imaging modality, target anatomy, pathology, and annotation structure. Continual learning allows models to adapt to new tasks without simultaneous access to previous datasets. However, when successive tasks differ substantially, learning a new task can degrade performance on earlier ones, a problem known as catastrophic forgetting. Understanding where adaptation disrupts previous knowledge can help guide the design of more targeted continual-learning strategies. We investigate how forgetting changes as different parts of an encoder--decoder network are allowed to adapt. We progressively expand the trainable region of a 3D nnU-Net backbone from the bottleneck toward input- and output-proximal blocks. Under a shared learning rate, adaptation near the bottleneck largely preserves previous-task performance but provides limited current-task learning, whereas broader adaptation improves current-task performance but sharply increases forgetting. This trade-off persists even when the average change in trainable backbone parameters is approximately comparable. Assigning different learning rates to different blocks substantially reduces forgetting when part of the backbone is trainable, although this changes both the size and location of the updates. Forgetting still increases as more blocks are trained and remains severe when the full backbone is updated. These results show that forgetting depends not only on how much the model changes, but also on which parts of the model are allowed to change.