# Federated Incremental Semantic Segmentation

arXiv: https://arxiv.org/abs/2304.04620 · HTML: https://mlsanity.com/paper/2304.04620/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 59 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Machine Learning: Concept and Applications](https://mlsanity.com/paper/1902.04885/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning](https://mlsanity.com/paper/2101.11203/) (2021)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Privacy-preserving Federated Brain Tumour Segmentation](https://mlsanity.com/paper/1910.00962/) (2019)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [Federated Adversarial Domain Adaptation](https://mlsanity.com/paper/1911.02054/) (2019)
- [Federated Class-Incremental Learning](https://mlsanity.com/paper/2203.11473/) (2022)
- [Is Out-of-Distribution Detection Learnable?](https://mlsanity.com/paper/2210.14707/) (2022)
- [Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation](https://mlsanity.com/paper/1903.07424/) (2019)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)

## Built on by (most cited first, 30 of 36)

- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [Point, Segment and Count: A Generalized Framework for Object Counting](https://mlsanity.com/paper/2311.12386/) (2023)
- [Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation](https://mlsanity.com/paper/2306.11087/) (2023)
- [Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey](https://mlsanity.com/paper/2412.13840/) (2024)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Personalized Federated Domain-Incremental Learning based on Adaptive Knowledge Matching](https://mlsanity.com/paper/2407.05005/) (2024)
- [Heterogeneous Forgetting Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2308.03374/) (2023)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Prototype Correlation Matching and Class-Relation Reasoning for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2406.05054/) (2024)
- [Federated Continual Learning: Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2502.07059/) (2025)
- [FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models](https://mlsanity.com/paper/2501.02461/) (2025)
- [Gradient-Semantic Compensation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2307.10822/) (2023)
- [Self-paced Weight Consolidation for Continual Learning](https://mlsanity.com/paper/2307.10845/) (2023)
- [Task Relation Distillation and Prototypical Pseudo Label for Incremental Named Entity Recognition](https://mlsanity.com/paper/2308.08793/) (2023)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [Class-wise Balancing Data Replay for Federated Class-Incremental Learning](https://mlsanity.com/paper/2507.07712/) (2025)
- [Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systems](https://mlsanity.com/paper/2604.20745/) (2026)
- [FediOS: Decoupling Orthogonal Subspaces for Personalization in Feature-skew Federated Learning](https://mlsanity.com/paper/2311.18559/) (2023)
- [Resource-Constrained Federated Continual Learning: What Does Matter?](https://mlsanity.com/paper/2501.08737/) (2025)
- [RECALL+: Adversarial Web-based Replay for Continual Learning in Semantic Segmentation](https://mlsanity.com/paper/2309.10479/) (2023)
- [Federated Continual Novel Class Learning](https://mlsanity.com/paper/2312.13500/) (2023)
- [Future-Proofing Class-Incremental Learning](https://mlsanity.com/paper/2404.03200/) (2024)
- [Rehearsal-free Federated Domain-incremental Learning](https://mlsanity.com/paper/2405.13900/) (2024)
- [Bridging Data Islands: Geographic Heterogeneity-Aware Federated Learning for Collaborative Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2404.09292/) (2024)
- [Federated Incremental Named Entity Recognition](https://mlsanity.com/paper/2411.11623/) (2024)
- [Energy-Efficient and Real-Time Sensing for Federated Continual Learning via Sample-Driven Control](https://mlsanity.com/paper/2310.07497/) (2023)
- [Variational Bayes for Federated Continual Learning](https://mlsanity.com/paper/2405.14291/) (2024)
- [Improving Generalization in Heterogeneous Federated Continual Learning via Spatio-Temporal Gradient Matching with Prototypical Coreset](https://mlsanity.com/paper/2506.12031/) (2025)
- [Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition](https://mlsanity.com/paper/2508.03259/) (2025)
- [Federated Learning with New Knowledge: Fundamentals, Advances, and Futures](https://mlsanity.com/paper/2402.02268/) (2024)

## Abstract (arXiv)

Federated learning-based semantic segmentation (FSS) has drawn widespread attention via decentralized training on local clients. However, most FSS models assume categories are fixed in advance, thus heavily undergoing forgetting on old categories in practical applications where local clients receive new categories incrementally while have no memory storage to access old classes. Moreover, new clients collecting novel classes may join in the global training of FSS, which further exacerbates catastrophic forgetting. To surmount the above challenges, we propose a Forgetting-Balanced Learning (FBL) model to address heterogeneous forgetting on old classes from both intra-client and inter-client aspects. Specifically, under the guidance of pseudo labels generated via adaptive class-balanced pseudo labeling, we develop a forgetting-balanced semantic compensation loss and a forgetting-balanced relation consistency loss to rectify intra-client heterogeneous forgetting of old categories with background shift. It performs balanced gradient propagation and relation consistency distillation within local clients. Moreover, to tackle heterogeneous forgetting from inter-client aspect, we propose a task transition monitor. It can identify new classes under privacy protection and store the latest old global model for relation distillation. Qualitative experiments reveal large improvement of our model against comparison methods. The code is available at https://github.com/JiahuaDong/FISS.