# Knowledge Distillation for Incremental Learning in Semantic Segmentation

arXiv: https://arxiv.org/abs/1911.03462 · HTML: https://mlsanity.com/paper/1911.03462/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2019.

## Built on by (most cited first, 30 of 40)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [RECALL: Replay-based Continual Learning in Semantic Segmentation](https://mlsanity.com/paper/2108.03673/) (2021)
- [Co-advise: Cross Inductive Bias Distillation](https://mlsanity.com/paper/2106.12378/) (2021)
- [Incremental Learning in Semantic Segmentation from Image Labels](https://mlsanity.com/paper/2112.01882/) (2021)
- [CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation](https://mlsanity.com/paper/2107.02408/) (2021)
- [Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2203.14098/) (2022)
- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [RBC: Rectifying the Biased Context in Continual Semantic Segmentation](https://mlsanity.com/paper/2203.08404/) (2022)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Monitored Distillation for Positive Congruent Depth Completion](https://mlsanity.com/paper/2203.16034/) (2022)
- [Unsupervised Continual Semantic Adaptation through Neural Rendering](https://mlsanity.com/paper/2211.13969/) (2022)
- [Online Continual Learning For Interactive Instruction Following Agents](https://mlsanity.com/paper/2403.07548/) (2024)
- [Distilling Efficient Vision Transformers from CNNs for Semantic Segmentation](https://mlsanity.com/paper/2310.07265/) (2023)
- [Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation](https://mlsanity.com/paper/2207.12964/) (2022)
- [Towards Unsupervised Open World Semantic Segmentation](https://mlsanity.com/paper/2201.01073/) (2022)
- [Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory](https://mlsanity.com/paper/2309.15413/) (2023)
- [Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data](https://mlsanity.com/paper/2112.12833/) (2021)
- [Learning with Style: Continual Semantic Segmentation Across Tasks and Domains](https://mlsanity.com/paper/2210.07016/) (2022)
- [Hybrid Open-set Segmentation with Synthetic Negative Data](https://mlsanity.com/paper/2301.08555/) (2023)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [RaSP: Relation-aware Semantic Prior for Weakly Supervised Incremental Segmentation](https://mlsanity.com/paper/2305.19879/) (2023)
- [Gradual Divergence for Seamless Adaptation: A Novel Domain Incremental Learning Method](https://mlsanity.com/paper/2406.16231/) (2024)
- [Modality-Incremental Learning with Disjoint Relevance Mapping Networks for Image-based Semantic Segmentation](https://mlsanity.com/paper/2411.17610/) (2024)
- [Self-Improving Semantic Perception for Indoor Localisation](https://mlsanity.com/paper/2105.01595/) (2021)
- [Test-time Adaptation in the Dynamic World with Compound Domain Knowledge Management](https://mlsanity.com/paper/2212.08356/) (2022)
- [Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation](https://mlsanity.com/paper/2110.09674/) (2021)
- [Detecting and Learning the Unknown in Semantic Segmentation](https://mlsanity.com/paper/2202.08700/) (2022)
- [RECALL+: Adversarial Web-based Replay for Continual Learning in Semantic Segmentation](https://mlsanity.com/paper/2309.10479/) (2023)
- [Primal Dual Continual Learning: Balancing Stability and Plasticity through Adaptive Memory Allocation](https://mlsanity.com/paper/2310.00154/) (2023)

## Abstract (arXiv)

Deep learning architectures have shown remarkable results in scene understanding problems, however they exhibit a critical drop of performances when they are required to learn incrementally new tasks without forgetting old ones. This catastrophic forgetting phenomenon impacts on the deployment of artificial intelligence in real world scenarios where systems need to learn new and different representations over time. Current approaches for incremental learning deal only with image classification and object detection tasks, while in this work we formally introduce incremental learning for semantic segmentation. We tackle the problem applying various knowledge distillation techniques on the previous model. In this way, we retain the information about learned classes, whilst updating the current model to learn the new ones. We developed four main methodologies of knowledge distillation working on both output layers and internal feature representations. We do not store any image belonging to previous training stages and only the last model is used to preserve high accuracy on previously learned classes. Extensive experimental results on the Pascal VOC2012 and MSRC-v2 datasets show the effectiveness of the proposed approaches in several incremental learning scenarios.