# SELECT: SELEctive Context Transfer for Class-Incremental Semantic Segmentation

arXiv: https://arxiv.org/abs/2608.30281 · HTML: https://mlsanity.com/paper/2608.30281/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 45 of its 62 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Builds on (most cited first, 30 of 45)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [Incremental Learning Through Deep Adaptation](https://mlsanity.com/paper/1705.04228/) (2017)
- [SegViT: Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2210.05844/) (2022)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning](https://mlsanity.com/paper/2106.11562/) (2021)
- [Deep Generative Dual Memory Network for Continual Learning](https://mlsanity.com/paper/1710.10368/) (2017)
- [Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2404.00781/) (2024)
- [RBC: Rectifying the Biased Context in Continual Semantic Segmentation](https://mlsanity.com/paper/2203.08404/) (2022)
- [Solving the Catastrophic Forgetting Problem in Generalized Category Discovery](https://mlsanity.com/paper/2501.05272/) (2025)
- [CoMFormer: Continual Learning in Semantic and Panoptic Segmentation](https://mlsanity.com/paper/2211.13999/) (2022)
- [Attribution-aware Weight Transfer: A Warm-Start Initialization for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.07207/) (2022)
- [OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2402.04129/) (2024)
- [ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning](https://mlsanity.com/paper/2403.20126/) (2024)
- [Disentangling and Mitigating the Impact of Task Similarity for Continual Learning](https://mlsanity.com/paper/2405.20236/) (2024)
- [CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation](https://mlsanity.com/paper/2310.06368/) (2023)
- [One-stage Prompt-based Continual Learning](https://mlsanity.com/paper/2402.16189/) (2024)
- [Rethinking Query-based Transformer for Continual Image Segmentation](https://mlsanity.com/paper/2507.07831/) (2025)

## Abstract (arXiv)

Class-Incremental Semantic Segmentation (CISS) is fundamentally challenged by catastrophic forgetting and background shift, where learning new concepts degrades performance on previously seen classes. While existing methods attempt to balance stability (retaining old knowledge) and plasticity (learning new knowledge), they often fail to leverage prior knowledge effectively. These approaches typically rely on indiscriminate knowledge transfer or ambiguous initializations, which can dilute crucial semantic information. To overcome this limitation, we propose SELECT, a novel approach for Selective Context Transfer, which instead grounds each new class in a small set of semantically similar past classes. Its core is a Context Transfer Attention mechanism that aggregates the learned tokens from similar classes into a structured initialization for the new class. To ensure this transfer does not corrupt the borrowed representations, we add a controlled noise perturbation and a margin-based context-transfer loss that enforces separation between the new class token and its source tokens. Extensive experiments on Pascal VOC and ADE20K show that SELECT consistently outperforms prior work, achieving mIoU of 2.2% on VOC and 2.8% on ADE, providing an effective handle on the stability-plasticity dilemma. Code is available at https://github.com/avigupta2798/SELECT.