# Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory

arXiv: https://arxiv.org/abs/2309.15413 · HTML: https://mlsanity.com/paper/2309.15413/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 79 of its 112 references that are in the map.
- First posted on arXiv 27 Sep 2023.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 79)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)

## Built on by (most cited first, 12 of 12)

- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling](https://mlsanity.com/paper/2407.15429/) (2024)
- [Background Adaptation with Residual Modeling for Exemplar-Free Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.09838/) (2024)
- [Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images](https://mlsanity.com/paper/2407.14242/) (2024)
- [Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting](https://mlsanity.com/paper/2406.19796/) (2024)
- [Decoupling Continual Semantic Segmentation](https://mlsanity.com/paper/2508.05065/) (2025)
- [Continual Contrastive Spoken Language Understanding](https://mlsanity.com/paper/2310.02699/) (2023)
- [Reconciling Semantic Controllability and Diversity for Remote Sensing Image Synthesis with Hybrid Semantic Embedding](https://mlsanity.com/paper/2411.14781/) (2024)
- [Evolving Without Ending: Unifying Multimodal Incremental Learning for Continual Panoptic Perception](https://mlsanity.com/paper/2601.15643/) (2026)
- [PILOT: A Data-Free Continual Learning Approach for Real-Time Semantic Segmentation via Boundary Guidance](https://mlsanity.com/paper/2605.27128/) (2026)
- [Efficient Remote Sensing Instance Segmentation with Linear-Time State Space Distilled Visual Foundation Models](https://mlsanity.com/paper/2606.25324/) (2026)
- [SELECT: SELEctive Context Transfer for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2608.30281/) (2026)

## Abstract (arXiv)

As a front-burner problem in incremental learning, class incremental semantic segmentation (CISS) is plagued by catastrophic forgetting and semantic drift. Although recent methods have utilized knowledge distillation to transfer knowledge from the old model, they are still unable to avoid pixel confusion, which results in severe misclassification after incremental steps due to the lack of annotations for past and future classes. Meanwhile data-replay-based approaches suffer from storage burdens and privacy concerns. In this paper, we propose to address CISS without exemplar memory and resolve catastrophic forgetting as well as semantic drift synchronously. We present Inherit with Distillation and Evolve with Contrast (IDEC), which consists of a Dense Knowledge Distillation on all Aspects (DADA) manner and an Asymmetric Region-wise Contrastive Learning (ARCL) module. Driven by the devised dynamic class-specific pseudo-labelling strategy, DADA distils intermediate-layer features and output-logits collaboratively with more emphasis on semantic-invariant knowledge inheritance. ARCL implements region-wise contrastive learning in the latent space to resolve semantic drift among known classes, current classes, and unknown classes. We demonstrate the effectiveness of our method on multiple CISS tasks by state-of-the-art performance, including Pascal VOC 2012, ADE20K and ISPRS datasets. Our method also shows superior anti-forgetting ability, particularly in multi-step CISS tasks.