# SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

arXiv: https://arxiv.org/abs/2106.11562 · HTML: https://mlsanity.com/paper/2106.11562/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup](https://mlsanity.com/paper/2009.06962/) (2020)
- [A Simple Pooling-Based Design for Real-Time Salient Object Detection](https://mlsanity.com/paper/1904.09569/) (2019)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [Deeply supervised salient object detection with short connections](https://mlsanity.com/paper/1611.04849/) (2016)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity](https://mlsanity.com/paper/2102.03065/) (2021)
- [SS-IL: Separated Softmax for Incremental Learning](https://mlsanity.com/paper/2003.13947/) (2020)
- [Salient Object Detection: A Discriminative Regional Feature Integration Approach](https://mlsanity.com/paper/1410.5926/) (2014)
- [Optimal Continual Learning has Perfect Memory and is NP-hard](https://mlsanity.com/paper/2006.05188/) (2020)
- [NSML: Meet the MLaaS platform with a real-world case study](https://mlsanity.com/paper/1810.09957/) (2018)
- [Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2103.07246/) (2021)
- [Saliency Guided Self-attention Network for Weakly and Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/1910.05475/) (2019)

## Built on by (most cited first, 30 of 71)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Continual Vision-Language Representation Learning with Off-Diagonal Information](https://mlsanity.com/paper/2305.07437/) (2023)
- [Continual Semantic Segmentation with Automatic Memory Sample Selection](https://mlsanity.com/paper/2304.05015/) (2023)
- [Federated Incremental Semantic Segmentation](https://mlsanity.com/paper/2304.04620/) (2023)
- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.05941/) (2022)
- [Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation](https://mlsanity.com/paper/2211.06866/) (2022)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [SATS: Self-Attention Transfer for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.07667/) (2022)
- [ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.06816/) (2022)
- [SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2306.06289/) (2023)
- [Attribution-aware Weight Transfer: A Warm-Start Initialization for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.07207/) (2022)
- [The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation](https://mlsanity.com/paper/2303.15062/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay](https://mlsanity.com/paper/2308.14100/) (2023)
- [ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning](https://mlsanity.com/paper/2403.20126/) (2024)
- [kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies](https://mlsanity.com/paper/2404.09447/) (2024)
- [CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation](https://mlsanity.com/paper/2310.06368/) (2023)
- [Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments](https://mlsanity.com/paper/2305.15700/) (2023)
- [Rethinking Query-based Transformer for Continual Image Segmentation](https://mlsanity.com/paper/2507.07831/) (2025)
- [FALCON: Fairness Learning via Contrastive Attention Approach to Continual Semantic Scene Understanding](https://mlsanity.com/paper/2311.15965/) (2023)
- [Subspace Distillation for Continual Learning](https://mlsanity.com/paper/2307.16419/) (2023)
- [Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory](https://mlsanity.com/paper/2309.15413/) (2023)
- [Continual Learning for Image Segmentation with Dynamic Query](https://mlsanity.com/paper/2311.17450/) (2023)
- [Hyperparameters in Continual Learning: A Reality Check](https://mlsanity.com/paper/2403.09066/) (2024)
- [Strike a Balance in Continual Panoptic Segmentation](https://mlsanity.com/paper/2407.16354/) (2024)
- [DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation](https://mlsanity.com/paper/2308.01127/) (2023)
- [Continual Segmentation with Disentangled Objectness Learning and Class Recognition](https://mlsanity.com/paper/2403.03477/) (2024)
- [Mitigating Background Shift in Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.11859/) (2024)

## Abstract (arXiv)

This paper introduces a solid state-of-the-art baseline for a class-incremental semantic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed to fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (1) defining unknown classes within the background class to help to learn future classes (help plasticity), (2) freezing backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https://github.com/clovaai/SSUL.