# Continual Attentive Fusion for Incremental Learning in Semantic Segmentation

arXiv: https://arxiv.org/abs/2202.00432 · HTML: https://mlsanity.com/paper/2202.00432/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 78 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Attention to Scale: Scale-aware Semantic Image Segmentation](https://mlsanity.com/paper/1511.03339/) (2015)

## Built on by (most cited first, 15 of 15)

- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning](https://mlsanity.com/paper/2301.10931/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Multi-Label Continual Learning using Augmented Graph Convolutional Network](https://mlsanity.com/paper/2211.14763/) (2022)
- [Gradient-Semantic Compensation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2307.10822/) (2023)
- [Learning with Style: Continual Semantic Segmentation Across Tasks and Domains](https://mlsanity.com/paper/2210.07016/) (2022)
- [Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning](https://mlsanity.com/paper/2503.17914/) (2025)
- [CLEO: Continual Learning of Evolving Ontologies](https://mlsanity.com/paper/2407.08411/) (2024)
- [Unified Open-Vocabulary Dense Visual Prediction](https://mlsanity.com/paper/2307.08238/) (2023)
- [Constructing Enhanced Mutual Information for Online Class-Incremental Learning](https://mlsanity.com/paper/2407.18526/) (2024)
- [Multimodal Data Storage and Retrieval for Embodied AI: A Survey](https://mlsanity.com/paper/2508.13901/) (2025)
- [Delving into Transformer for Incremental Semantic Segmentation](https://mlsanity.com/paper/2211.10253/) (2022)
- [Hierarchical Cross-Attention Network for Virtual Try-On](https://mlsanity.com/paper/2411.15542/) (2024)
- [Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning](https://mlsanity.com/paper/2605.08839/) (2026)
- [Generalization-aware Remote Sensing Change Detection via Domain-agnostic Learning](https://mlsanity.com/paper/2504.00543/) (2025)

## Abstract (arXiv)

Over the past years, semantic segmentation, as many other tasks in computer vision, benefited from the progress in deep neural networks, resulting in significantly improved performance. However, deep architectures trained with gradient-based techniques suffer from catastrophic forgetting, which is the tendency to forget previously learned knowledge while learning new tasks. Aiming at devising strategies to counteract this effect, incremental learning approaches have gained popularity over the past years. However, the first incremental learning methods for semantic segmentation appeared only recently. While effective, these approaches do not account for a crucial aspect in pixel-level dense prediction problems, i.e. the role of attention mechanisms. To fill this gap, in this paper we introduce a novel attentive feature distillation approach to mitigate catastrophic forgetting while accounting for semantic spatial- and channel-level dependencies. Furthermore, we propose a {continual attentive fusion} structure, which takes advantage of the attention learned from the new and the old tasks while learning features for the new task. Finally, we also introduce a novel strategy to account for the background class in the distillation loss, thus preventing biased predictions. We demonstrate the effectiveness of our approach with an extensive evaluation on Pascal-VOC 2012 and ADE20K, setting a new state of the art.