# Federated Continual Learning via Knowledge Fusion: A Survey

arXiv: https://arxiv.org/abs/2312.16475 · HTML: https://mlsanity.com/paper/2312.16475/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 92 of its 145 references that are in the map.
- First posted on arXiv 27 Dec 2023.

## Builds on (most cited first, 30 of 92)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)

## Built on by (most cited first, 30 of 53)

- [Recent Advances of Multimodal Continual Learning: A Comprehensive Survey](https://mlsanity.com/paper/2410.05352/) (2024)
- [Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning](https://mlsanity.com/paper/2409.01128/) (2024)
- [Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey](https://mlsanity.com/paper/2412.13840/) (2024)
- [Ten Challenging Problems in Federated Foundation Models](https://mlsanity.com/paper/2502.12176/) (2025)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Unlearning through Knowledge Overwriting: Reversible Federated Unlearning via Selective Sparse Adapter](https://mlsanity.com/paper/2502.20709/) (2025)
- [Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence](https://mlsanity.com/paper/2412.13779/) (2024)
- [Synergizing Foundation Models and Federated Learning: A Survey](https://mlsanity.com/paper/2406.12844/) (2024)
- [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024)
- [Federated Continual Learning: Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2502.07059/) (2025)
- [Federated Continual Instruction Tuning](https://mlsanity.com/paper/2503.12897/) (2025)
- [Federated Continual Learning for Edge-AI: A Comprehensive Survey](https://mlsanity.com/paper/2411.13740/) (2024)
- [FedProK: Trustworthy Federated Class-Incremental Learning via Prototypical Feature Knowledge Transfer](https://mlsanity.com/paper/2405.02685/) (2024)
- [Handling Spatial-Temporal Data Heterogeneity for Federated Continual Learning via Tail Anchor](https://mlsanity.com/paper/2412.18355/) (2024)
- [Class-wise Balancing Data Replay for Federated Class-Incremental Learning](https://mlsanity.com/paper/2507.07712/) (2025)
- [SacFL: Self-Adaptive Federated Continual Learning for Resource-Constrained End Devices](https://mlsanity.com/paper/2505.00365/) (2025)
- [Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning](https://mlsanity.com/paper/2506.05977/) (2025)
- [Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systems](https://mlsanity.com/paper/2604.20745/) (2026)
- [UniSD: Towards a Unified Self-Distillation Framework for Large Language Models](https://mlsanity.com/paper/2605.06597/) (2026)
- [On the Convergence of Continual Federated Learning Using Incrementally Aggregated Gradients](https://mlsanity.com/paper/2411.07959/) (2024)
- [Resource-Constrained Federated Continual Learning: What Does Matter?](https://mlsanity.com/paper/2501.08737/) (2025)
- [A Thorough Assessment of the Non-IID Data Impact in Federated Learning](https://mlsanity.com/paper/2503.17070/) (2025)
- [Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator](https://mlsanity.com/paper/2407.11078/) (2024)
- [Memory Bank Compression for Continual Adaptation of Large Language Models](https://mlsanity.com/paper/2601.00756/) (2026)
- [AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Data](https://mlsanity.com/paper/2505.12245/) (2025)
- [ErrorEraser: Unlearning Data Bias for Improved Continual Learning](https://mlsanity.com/paper/2506.09347/) (2025)
- [Decentralized Dynamic Cooperation of Personalized Models for Federated Continual Learning](https://mlsanity.com/paper/2509.23683/) (2025)
- [ConSurv: Multimodal Continual Learning for Survival Analysis](https://mlsanity.com/paper/2511.09853/) (2025)
- [FedRewind: Rewinding Continual Model Exchange for Decentralized Federated Learning](https://mlsanity.com/paper/2411.09842/) (2024)
- [Orthogonal Calibration for Asynchronous Federated Learning](https://mlsanity.com/paper/2502.15940/) (2025)

## Abstract (arXiv)

Data privacy and silos are nontrivial and greatly challenging in many real-world applications. Federated learning is a decentralized approach to training models across multiple local clients without the exchange of raw data from client devices to global servers. However, existing works focus on a static data environment and ignore continual learning from streaming data with incremental tasks. Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments. The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones. In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion. In summary, our motivations are four-fold: we (1) raise a fundamental problem called ''spatial-temporal catastrophic forgetting'' and evaluate its impact on the performance using a well-known method called federated averaging (FedAvg), (2) integrate most of the existing FCL methods into two generic frameworks, namely synchronous FCL and asynchronous FCL, (3) categorize a large number of methods according to the mechanism involved in knowledge fusion, and finally (4) showcase an outlook on the future work of FCL.