Cyclical Weight Consolidation: Towards Solving Catastrophic Forgetting in Serial Federated Learning
Not on the map. Why?
Abstract (arXiv)
Federated Learning (FL) has gained attention for addressing data scarcity and privacy concerns. While parallel FL algorithms like FedAvg exhibit remarkable performance, they face challenges in scenarios with diverse network speeds and concerns about centralized control, especially in multi-institutional collaborations like the medical domain. Serial FL presents an alternative solution, circumventing these challenges by transferring model updates serially between devices in a cyclical manner. Nevertheless, it is deemed inferior to parallel FL in that (1) its performance shows undesirable fluctuations, and (2) it converges to a lower plateau, particularly when dealing with non-IID data. The observed phenomenon is attributed to catastrophic forgetting due to knowledge loss from previous sites. In this paper, to overcome fluctuation and low efficiency in the iterative learning and forgetting process, we introduce cyclical weight consolidation (CWC), a straightforward yet potent approach specifically tailored for serial FL. CWC employs a consolidation matrix to regulate local optimization. This matrix tracks the significance of each parameter on the overall federation throughout the entire training trajectory, preventing abrupt changes in significant weights. During revisitation, to maintain adaptability, old memory undergoes decay to incorporate new information. Our comprehensive evaluations demonstrate that in various non-IID settings, CWC mitigates the fluctuation behavior of the original serial FL approach and enhances the converged performance consistently and significantly. The improved performance is either comparable to or better than the parallel vanilla.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 17 May 2024.
Builds on
23 of 42 references.
References · 23
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
- 2016Progressive Neural Networkscited by 1,361
- 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
- 2018Experience Replay for Continual Learningcited by 1,095
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
- 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 669
- 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
11 more
- 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
- 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
- 2019On Tiny Episodic Memories in Continual Learningcited by 307
- 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
- 2018Selective Experience Replay for Lifelong Learningcited by 241
- 2018Task-Free Continual Learningcited by 221
- 2018Reinforced Continual Learningcited by 186
- 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 149
- 2021Decentralized Federated Learning through Proxy Model Sharingcited by 57
- 2022Personalizing Federated Medical Image Segmentation via Local Calibrationcited by 29
- 2021Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillationcited by 15
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.