# Continual Local Training for Better Initialization of Federated Models

arXiv: https://arxiv.org/abs/2005.12657 · HTML: https://mlsanity.com/paper/2005.12657/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2020.

## Built on by (most cited first, 30 of 34)

- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Federated Continual Learning via Knowledge Fusion: A Survey](https://mlsanity.com/paper/2312.16475/) (2023)
- [A distillation-based approach integrating continual learning and federated learning for pervasive services](https://mlsanity.com/paper/2109.04197/) (2021)
- [Asynchronous Federated Continual Learning](https://mlsanity.com/paper/2304.03626/) (2023)
- [Privacy and Fairness in Federated Learning: on the Perspective of Trade-off](https://mlsanity.com/paper/2306.14123/) (2023)
- [Federated Learning from Only Unlabeled Data with Class-Conditional-Sharing Clients](https://mlsanity.com/paper/2204.03304/) (2022)
- [SphereFed: Hyperspherical Federated Learning](https://mlsanity.com/paper/2207.09413/) (2022)
- [FedBone: Towards Large-Scale Federated Multi-Task Learning](https://mlsanity.com/paper/2306.17465/) (2023)
- [Spatial-Temporal Federated Learning for Lifelong Person Re-identification on Distributed Edges](https://mlsanity.com/paper/2207.11759/) (2022)
- [Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations](https://mlsanity.com/paper/2405.09839/) (2024)
- [FedKNOW: Federated Continual Learning with Signature Task Knowledge Integration at Edge](https://mlsanity.com/paper/2212.01738/) (2022)
- [BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activities](https://mlsanity.com/paper/2501.14309/) (2025)
- [Continual Horizontal Federated Learning for Heterogeneous Data](https://mlsanity.com/paper/2203.02108/) (2022)
- [A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications](https://mlsanity.com/paper/2403.01387/) (2024)
- [Free Lunch for Federated Remote Sensing Target Fine-Grained Classification: A Parameter-Efficient Framework](https://mlsanity.com/paper/2401.01493/) (2024)
- [Jointly Exploring Client Drift and Catastrophic Forgetting in Dynamic Learning](https://mlsanity.com/paper/2309.00688/) (2023)
- [FedJudge: Federated Legal Large Language Model](https://mlsanity.com/paper/2309.08173/) (2023)
- [Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments](https://mlsanity.com/paper/2110.11006/) (2021)
- [On Knowledge Editing in Federated Learning: Perspectives, Challenges, and Future Directions](https://mlsanity.com/paper/2306.01431/) (2023)
- [A Survey of Incremental Transfer Learning: Combining Peer-to-Peer Federated Learning and Domain Incremental Learning for Multicenter Collaboration](https://mlsanity.com/paper/2309.17192/) (2023)
- [Federated Distillation for Medical Image Classification: Towards Trustworthy Computer-Aided Diagnosis](https://mlsanity.com/paper/2407.02261/) (2024)
- [Coordinated Replay Sample Selection for Continual Federated Learning](https://mlsanity.com/paper/2310.15054/) (2023)
- [Global Update Guided Federated Learning](https://mlsanity.com/paper/2204.03920/) (2022)
- [FedMeS: Personalized Federated Continual Learning Leveraging Local Memory](https://mlsanity.com/paper/2404.12710/) (2024)
- [Buffer-based Gradient Projection for Continual Federated Learning](https://mlsanity.com/paper/2409.01585/) (2024)
- [FedRewind: Rewinding Continual Model Exchange for Decentralized Federated Learning](https://mlsanity.com/paper/2411.09842/) (2024)
- [Towards More Efficient Federated Learning with Better Optimization Objects](https://mlsanity.com/paper/2108.08577/) (2021)
- [Exploration and Exploitation in Federated Learning to Exclude Clients with Poisoned Data](https://mlsanity.com/paper/2204.14020/) (2022)
- [Federated Learning with New Knowledge: Fundamentals, Advances, and Futures](https://mlsanity.com/paper/2402.02268/) (2024)

## Abstract (arXiv)

Federated learning (FL) refers to the learning paradigm that trains machine learning models directly in the decentralized systems consisting of smart edge devices without transmitting the raw data, which avoids the heavy communication costs and privacy concerns. Given the typical heterogeneous data distributions in such situations, the popular FL algorithm Federated Averaging (FedAvg) suffers from weight divergence and thus cannot achieve a competitive performance for the global model (denoted as the initial performance in FL) compared to centralized methods. In this paper, we propose the local continual training strategy to address this problem. Importance weights are evaluated on a small proxy dataset on the central server and then used to constrain the local training. With this additional term, we alleviate the weight divergence and continually integrate the knowledge on different local clients into the global model, which ensures a better generalization ability. Experiments on various FL settings demonstrate that our method significantly improves the initial performance of federated models with few extra communication costs.