# ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation

arXiv: https://arxiv.org/abs/2210.06816 · HTML: https://mlsanity.com/paper/2210.06816/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Deeply supervised salient object detection with short connections](https://mlsanity.com/paper/1611.04849/) (2016)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [Strip Pooling: Rethinking Spatial Pooling for Scene Parsing](https://mlsanity.com/paper/2003.13328/) (2020)
- [Memory-Efficient Incremental Learning Through Feature Adaptation](https://mlsanity.com/paper/2004.00713/) (2020)
- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [RECALL: Replay-based Continual Learning in Semantic Segmentation](https://mlsanity.com/paper/2108.03673/) (2021)
- [In-Place Activated BatchNorm for Memory-Optimized Training of DNNs](https://mlsanity.com/paper/1712.02616/) (2017)
- [SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning](https://mlsanity.com/paper/2106.11562/) (2021)

## Built on by (most cited first, 25 of 25)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Make Continual Learning Stronger via C-Flat](https://mlsanity.com/paper/2404.00986/) (2024)
- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies](https://mlsanity.com/paper/2404.09447/) (2024)
- [Gradient-Semantic Compensation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2307.10822/) (2023)
- [Self-paced Weight Consolidation for Continual Learning](https://mlsanity.com/paper/2307.10845/) (2023)
- [DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation](https://mlsanity.com/paper/2308.01127/) (2023)
- [Cs2K: Class-specific and Class-shared Knowledge Guidance for Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.09047/) (2024)
- [Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.14142/) (2024)
- [CoMBO: Conflict Mitigation via Branched Optimization for Class Incremental Segmentation](https://mlsanity.com/paper/2504.04156/) (2025)
- [Background Adaptation with Residual Modeling for Exemplar-Free Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.09838/) (2024)
- [Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration](https://mlsanity.com/paper/2403.11373/) (2024)
- [Flashbacks to Harmonize Stability and Plasticity in Continual Learning](https://mlsanity.com/paper/2506.00477/) (2025)
- [CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation](https://mlsanity.com/paper/2507.19887/) (2025)
- [C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning](https://mlsanity.com/paper/2508.18860/) (2025)
- [A Faster Path to Continual Learning](https://mlsanity.com/paper/2604.11064/) (2026)
- [Evolving Knowledge Mining for Class Incremental Segmentation](https://mlsanity.com/paper/2306.02027/) (2023)
- [Revisiting Continual Semantic Segmentation with Pre-trained Vision Models](https://mlsanity.com/paper/2508.04267/) (2025)
- [Incremental Human-Object Interaction Detection with Invariant Relation Representation Learning](https://mlsanity.com/paper/2510.27020/) (2025)
- [SAILS: Segment Anything with Incrementally Learned Semantics for Task-Invariant and Training-Free Continual Learning](https://mlsanity.com/paper/2602.14767/) (2026)
- [Learning Probabilistic Prompt for Continual Learning](https://mlsanity.com/paper/2607.04711/) (2026)
- [Knowing Beyond the Known: Reinforced Knowledge Specification for Multi-Label Class-Incremental Learning](https://mlsanity.com/paper/2608.30316/) (2026)

## Abstract (arXiv)

We address the problem of incremental semantic segmentation (ISS) recognizing novel object/stuff categories continually without forgetting previous ones that have been learned. The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time. To address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels. While such techniques are effective, there is still a lack of theoretical understanding of them. Replay-based methods propose to memorize a small set of images for previous categories. They achieve state-of-the-art performance at the cost of large memory footprint. We propose in this paper a novel ISS method, dubbed ALIFE, that provides a better compromise between accuracy and efficiency. To this end, we first show an in-depth analysis on the calibration techniques to better understand the effects on ISS. Based on this, we then introduce an adaptive logit regularizer (ALI) that enables our model to better learn new categories, while retaining knowledge for previous ones. We also present a feature replay scheme that memorizes features, instead of images directly, in order to reduce memory requirements significantly. Since a feature extractor is changed continually, memorized features should also be updated at every incremental stage. To handle this, we introduce category-specific rotation matrices updating the features for each category separately. We demonstrate the effectiveness of our approach with extensive experiments on standard ISS benchmarks, and show that our method achieves a better trade-off in terms of accuracy and efficiency.