mlsanityData to 14 Oct 2026

Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey

arXiv 2412.13840 ↗First posted 18 Dec 2024cs.LG cs.DCcited by 23

Not on the map. Why?

Abstract (arXiv)

Non-Centralized Continual Learning (NCCL) has become an emerging paradigm for enabling distributed devices such as vehicles and servers to handle streaming data from a joint non-stationary environment. To achieve high reliability and scalability in deploying this paradigm in distributed systems, it is essential to conquer challenges stemming from both spatial and temporal dimensions, manifesting as distribution shifts, catastrophic forgetting, heterogeneity, and privacy issues. This survey focuses on a comprehensive examination of the development of the non-centralized continual learning algorithms and the real-world deployment across distributed devices. We begin with an introduction to the background and fundamentals of non-centralized learning and continual learning. Then, we review existing solutions from three levels to represent how existing techniques alleviate the catastrophic forgetting and distribution shift. Additionally, we delve into the various types of heterogeneity issues, security, and privacy attributes, as well as real-world applications across three prevalent scenarios. Furthermore, we establish a large-scale benchmark to revisit this problem and analyze the performance of the state-of-the-art NCCL approaches. Finally, we discuss the important challenges and future research directions in NCCL.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 112 of its 267 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

112 of 267 references.

References · 112

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2016Learning without Forgettingcited by 2,496
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  10. 2016Progressive Neural Networkscited by 1,361
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
38 more
  1. 2018Experience Replay for Continual Learningcited by 1,095
  2. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  3. 2021Scaling Vision Transformerscited by 999
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  5. 2021Model-Contrastive Federated Learningcited by 766
  6. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  8. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
  9. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  10. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  11. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  12. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  13. 2021Gradient Projection Memory for Continual Learningcited by 348
  14. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  15. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  16. 2021Vision Transformers are Robust Learnerscited by 207
  17. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  18. 2019A Survey on Distributed Machine Learningcited by 188
  19. 2018Reinforced Continual Learningcited by 186
  20. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 161
  21. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  22. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  23. 2020A Survey on Conversational Recommender Systemscited by 143
  24. 2020Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challengescited by 139
  25. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 138
  26. 2022Federated Class-Incremental Learningcited by 127
  27. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  28. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 113
  29. 2017Incremental Learning Through Deep Adaptationcited by 108
  30. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 104
  31. 2019Reliable Federated Learning for Mobile Networkscited by 92
  32. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 78
  33. 2023Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Servicescited by 76
  34. 2020HFEL: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learningcited by 74
  35. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  36. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 65
  37. 2020Data Poisoning Attacks on Federated Machine Learningcited by 62
  38. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62

50 most cited of 112 All 112 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 9.

23 papers.

Sort:

By area:

  1. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  2. 2024Handling Spatial-Temporal Data Heterogeneity for Federated Continual Learning via Tail Anchorcited by 9
  3. 2026Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systemscited by 7
  4. 2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 6
  5. 2025Resource-Constrained Federated Continual Learning: What Does Matter?cited by 5
  6. 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learningcited by 5
  7. 2026Lifelong Embodied Navigation Learningcited by 3
  8. 2025An Attention-based Feature Memory Design for Energy-Efficient Continual Learningcited by 2
  9. 2025Geometric Prior-Guided Federated Prompt Calibrationcited by 2
  10. 2026All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptationcited by 2
  11. 2024Better Knowledge Enhancement for Privacy-Preserving Cross-Project Defect Predictioncited by 1
  12. 2024FedGIG: Graph Inversion from Gradient in Federated Learningcited by 1
11 more
  1. 2025Personalized Recommendation Models in Federated Settings: A Surveycited by 1
  2. 2025Onboard Optimization and Learning: A Surveycited by 1
  3. 2025Enhancing Federated Class-Incremental Learning via Spatial-Temporal Statistics Aggregationcited by 1
  4. 2025FedBiF: Communication-Efficient Federated Learning via Bits Freezingcited by 1
  5. 2026DeepAFL: Deep Analytic Federated Learningcited by 1
  6. 2025Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learningcited by 0
  7. 2026GAPSL: A Gradient-Aligned Parallel Split Learning over Data-Heterogeneous Edge Computing Systemscited by 0
  8. 2026Unbiased Rectification for Sequential Recommender Systems Under Fake Orderscited by 0
  9. 2026From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneitycited by 0
  10. 2026Distributed Optimization with Streaming Data: A Temporal Weighting Perspectivecited by 0
  11. 2026Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learningcited by 0