# Decoupling Continual Semantic Segmentation

arXiv: https://arxiv.org/abs/2508.05065 · HTML: https://mlsanity.com/paper/2508.05065/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 70 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 30 of 48)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Semantic Drift Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2004.00440/) (2020)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [S-LoRA: Serving Thousands of Concurrent LoRA Adapters](https://mlsanity.com/paper/2311.03285/) (2023)
- [RECALL: Replay-based Continual Learning in Semantic Segmentation](https://mlsanity.com/paper/2108.03673/) (2021)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning](https://mlsanity.com/paper/2106.11562/) (2021)
- [Open-Set Image Tagging with Multi-Grained Text Supervision](https://mlsanity.com/paper/2310.15200/) (2023)
- [Pin the Memory: Learning to Generalize Semantic Segmentation](https://mlsanity.com/paper/2204.03609/) (2022)
- [Dynamic Prototype Convolution Network for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2204.10638/) (2022)
- [Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data](https://mlsanity.com/paper/1810.12448/) (2018)
- [Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models](https://mlsanity.com/paper/2308.11186/) (2023)
- [Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.05941/) (2022)
- [Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation](https://mlsanity.com/paper/2211.06866/) (2022)
- [What is Wrong with Continual Learning in Medical Image Segmentation?](https://mlsanity.com/paper/2010.11008/) (2020)
- [SATS: Self-Attention Transfer for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.07667/) (2022)
- [CoMFormer: Continual Learning in Semantic and Panoptic Segmentation](https://mlsanity.com/paper/2211.13999/) (2022)
- [Semi-supervised Domain Adaptation via Sample-to-Sample Self-Distillation](https://mlsanity.com/paper/2111.14353/) (2021)
- [Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans](https://mlsanity.com/paper/2302.00162/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)

## Built on by (most cited first, 2 of 2)

- [Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration](https://mlsanity.com/paper/2511.15351/) (2025)
- [LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger](https://mlsanity.com/paper/2607.28374/) (2026)

## Abstract (arXiv)

Continual Semantic Segmentation (CSS) requires learning new classes without forgetting previously acquired knowledge, addressing the fundamental challenge of catastrophic forgetting in dense prediction tasks. However, existing CSS methods typically employ single-stage encoder-decoder architectures where segmentation masks and class labels are tightly coupled, leading to interference between old and new class learning and suboptimal retention-plasticity balance. We introduce DecoupleCSS, a novel two-stage framework for CSS. By decoupling class-aware detection from class-agnostic segmentation, DecoupleCSS enables more effective continual learning, preserving past knowledge while learning new classes. The first stage leverages pre-trained text and image encoders, adapted using LoRA, to encode class-specific information and generate location-aware prompts. In the second stage, the Segment Anything Model (SAM) is employed to produce precise segmentation masks, ensuring that segmentation knowledge is shared across both new and previous classes. This approach improves the balance between retention and adaptability in CSS, achieving state-of-the-art performance across a variety of challenging tasks. Our code is publicly available at: https://github.com/euyis1019/Decoupling-Continual-Semantic-Segmentation.