# Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey

arXiv: https://arxiv.org/abs/2412.13840 · HTML: https://mlsanity.com/paper/2412.13840/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 112 of its 267 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 30 of 112)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Federated Learning in Mobile Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/1909.11875/) (2019)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [A Unified Theory of Decentralized SGD with Changing Topology and Local Updates](https://mlsanity.com/paper/2003.10422/) (2020)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [Vision Transformers are Robust Learners](https://mlsanity.com/paper/2105.07581/) (2021)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [A Survey on Distributed Machine Learning](https://mlsanity.com/paper/1912.09789/) (2019)

## Built on by (most cited first, 23 of 23)

- [LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models](https://mlsanity.com/paper/2503.16843/) (2025)
- [Handling Spatial-Temporal Data Heterogeneity for Federated Continual Learning via Tail Anchor](https://mlsanity.com/paper/2412.18355/) (2024)
- [Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systems](https://mlsanity.com/paper/2604.20745/) (2026)
- [Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation](https://mlsanity.com/paper/2510.23123/) (2025)
- [Resource-Constrained Federated Continual Learning: What Does Matter?](https://mlsanity.com/paper/2501.08737/) (2025)
- [Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning](https://mlsanity.com/paper/2507.10348/) (2025)
- [Lifelong Embodied Navigation Learning](https://mlsanity.com/paper/2603.06073/) (2026)
- [An Attention-based Feature Memory Design for Energy-Efficient Continual Learning](https://mlsanity.com/paper/2510.04660/) (2025)
- [Geometric Prior-Guided Federated Prompt Calibration](https://mlsanity.com/paper/2512.07208/) (2025)
- [All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation](https://mlsanity.com/paper/2603.14276/) (2026)
- [Better Knowledge Enhancement for Privacy-Preserving Cross-Project Defect Prediction](https://mlsanity.com/paper/2412.17317/) (2024)
- [FedGIG: Graph Inversion from Gradient in Federated Learning](https://mlsanity.com/paper/2412.18513/) (2024)
- [Personalized Recommendation Models in Federated Settings: A Survey](https://mlsanity.com/paper/2504.07101/) (2025)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [Enhancing Federated Class-Incremental Learning via Spatial-Temporal Statistics Aggregation](https://mlsanity.com/paper/2506.01327/) (2025)
- [FedBiF: Communication-Efficient Federated Learning via Bits Freezing](https://mlsanity.com/paper/2509.10161/) (2025)
- [DeepAFL: Deep Analytic Federated Learning](https://mlsanity.com/paper/2603.00579/) (2026)
- [Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learning](https://mlsanity.com/paper/2504.19103/) (2025)
- [GAPSL: A Gradient-Aligned Parallel Split Learning over Data-Heterogeneous Edge Computing Systems](https://mlsanity.com/paper/2603.18540/) (2026)
- [Unbiased Rectification for Sequential Recommender Systems Under Fake Orders](https://mlsanity.com/paper/2604.08550/) (2026)
- [From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity](https://mlsanity.com/paper/2604.08617/) (2026)
- [Distributed Optimization with Streaming Data: A Temporal Weighting Perspective](https://mlsanity.com/paper/2608.09565/) (2026)
- [Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning](https://mlsanity.com/paper/2609.29711/) (2026)

## Abstract (arXiv)

Non-Centralized Continual Learning (NCCL) has become an emerging paradigm for enabling distributed devices such as vehicles and servers to handle streaming data from a joint non-stationary environment. To achieve high reliability and scalability in deploying this paradigm in distributed systems, it is essential to conquer challenges stemming from both spatial and temporal dimensions, manifesting as distribution shifts, catastrophic forgetting, heterogeneity, and privacy issues. This survey focuses on a comprehensive examination of the development of the non-centralized continual learning algorithms and the real-world deployment across distributed devices. We begin with an introduction to the background and fundamentals of non-centralized learning and continual learning. Then, we review existing solutions from three levels to represent how existing techniques alleviate the catastrophic forgetting and distribution shift. Additionally, we delve into the various types of heterogeneity issues, security, and privacy attributes, as well as real-world applications across three prevalent scenarios. Furthermore, we establish a large-scale benchmark to revisit this problem and analyze the performance of the state-of-the-art NCCL approaches. Finally, we discuss the important challenges and future research directions in NCCL.