# Continual Hippocampus Segmentation with Transformers

arXiv: https://arxiv.org/abs/2204.08043 · HTML: https://mlsanity.com/paper/2204.08043/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 17 Apr 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Vision Transformer for Small-Size Datasets](https://mlsanity.com/paper/2112.13492/) (2021)
- [Don't forget, there is more than forgetting: new metrics for Continual Learning](https://mlsanity.com/paper/1810.13166/) (2018)
- [Convolution-Free Medical Image Segmentation using Transformers](https://mlsanity.com/paper/2102.13645/) (2021)
- [Automated Design of Deep Learning Methods for Biomedical Image Segmentation](https://mlsanity.com/paper/1904.08128/) (2019)
- [What is Wrong with Continual Learning in Medical Image Segmentation?](https://mlsanity.com/paper/2010.11008/) (2020)
- [Towards continual learning in medical imaging](https://mlsanity.com/paper/1811.02496/) (2018)
- [AA-TransUNet: Attention Augmented TransUNet For Nowcasting Tasks](https://mlsanity.com/paper/2202.04996/) (2022)
- [Importance Driven Continual Learning for Segmentation Across Domains](https://mlsanity.com/paper/2005.00079/) (2020)

## Built on by (most cited first, 11 of 11)

- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Continual Learning in Medical Imaging: A Survey and Practical Analysis](https://mlsanity.com/paper/2405.13482/) (2024)
- [Generative appearance replay for continual unsupervised domain adaptation](https://mlsanity.com/paper/2301.01211/) (2023)
- [Continual atlas-based segmentation of prostate MRI](https://mlsanity.com/paper/2311.00548/) (2023)
- [Distribution-Aware Replay for Continual MRI Segmentation](https://mlsanity.com/paper/2407.21216/) (2024)
- [Detecting respiratory motion artefacts for cardiovascular MRIs to ensure high-quality segmentation](https://mlsanity.com/paper/2209.09678/) (2022)
- [NCAdapt: Dynamic adaptation with domain-specific Neural Cellular Automata for continual hippocampus segmentation](https://mlsanity.com/paper/2410.23368/) (2024)
- [Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge](https://mlsanity.com/paper/2507.19165/) (2025)
- [A Novel Hybrid Parameter-Efficient Fine-Tuning Approach for Hippocampus Segmentation and Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2409.00884/) (2024)
- [What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentation](https://mlsanity.com/paper/2608.13660/) (2026)

## Abstract (arXiv)

In clinical settings, where acquisition conditions and patient populations change over time, continual learning is key for ensuring the safe use of deep neural networks. Yet most existing work focuses on convolutional architectures and image classification. Instead, radiologists prefer to work with segmentation models that outline specific regions-of-interest, for which Transformer-based architectures are gaining traction. The self-attention mechanism of Transformers could potentially mitigate catastrophic forgetting, opening the way for more robust medical image segmentation. In this work, we explore how recently-proposed Transformer mechanisms for semantic segmentation behave in sequential learning scenarios, and analyse how best to adapt continual learning strategies for this setting. Our evaluation on hippocampus segmentation shows that Transformer mechanisms mitigate catastrophic forgetting for medical image segmentation compared to purely convolutional architectures, and demonstrates that regularising ViT modules should be done with caution.