mlsanityData to 7 Oct 2026

Acceleration of Federated Learning with Alleviated Forgetting in Local Training

arXiv 2203.02645 ↗First posted 5 Mar 2022cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy by independently training local models on each client and then aggregating parameters on a central server, thereby producing an effective global model. Although a variety of FL algorithms have been proposed, their training efficiency remains low when the data are not independently and identically distributed (non-i.i.d.) across different clients. We observe that the slow convergence rates of the existing methods are (at least partially) caused by the catastrophic forgetting issue during the local training stage on each individual client, which leads to a large increase in the loss function concerning the previous training data at the other clients. Here, we propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage by regularizing locally trained parameters with the loss on generated pseudo data, which encode the knowledge of previous training data learned by the global model. Our comprehensive experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep and the clients' data are extremely non-i.i.d., but is also able to protect privacy better in classification problems and more robust against gradient inversion attacks. The code is available at: https://github.com/Zoesgithub/FedReg.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 5 Mar 2022.

Builds on

33 of 48 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  12. 2018Group Normalizationcited by 1,514
21 more
  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  2. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  3. 2018Federated Learning with Non-IID Datacited by 1,099
  4. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  5. 2019Deep Leakage from Gradientscited by 1,004
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  7. 2021Model-Contrastive Federated Learningcited by 762
  8. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  9. 2020Federated Learning with Matched Averagingcited by 564
  10. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  11. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  12. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  13. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  14. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  15. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
  16. 2020Federated Visual Classification with Real-World Data Distributioncited by 126
  17. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  18. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  19. 2021Personalized Federated Learning with Gaussian Processescited by 76
  20. 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
  21. 2020Federated Learning via Synthetic Datacited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 12; 2024: 7; 2025: 4; 2026: 1.

29 papers.

Sort:

  1. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  2. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  3. 2023Personalized Federated Learning under Mixture of Distributionscited by 37
  4. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  5. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  6. 2022Meta Knowledge Condensation for Federated Learningcited by 26
  7. 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
  8. 2022Accelerated Federated Learning with Decoupled Adaptive Optimizationcited by 23
  9. 2022SphereFed: Hyperspherical Federated Learningcited by 19
  10. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  11. 2023Feature Matching Data Synthesis for Non-IID Federated Learningcited by 13
  12. 2023Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learningcited by 9
17 more
  1. 2023Towards Instance-adaptive Inference for Federated Learningcited by 7
  2. 2023Fake It Till Make It: Federated Learning with Consensus-Oriented Generationcited by 6
  3. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learningcited by 5
  4. 2024Flashback: Understanding and Mitigating Forgetting in Federated Learningcited by 4
  5. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Datacited by 4
  6. 2024FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learningcited by 3
  7. 2024An Element-Wise Weights Aggregation Method for Federated Learningcited by 3
  8. 2022FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularizationcited by 2
  9. 2025pMixFed: Efficient Personalized Federated Learning through Adaptive Layer-Wise Mixupcited by 2
  10. 2025Data Heterogeneity and Forgotten Labels in Split Federated Learningcited by 2
  11. 2023Structured Cooperative Learning with Graphical Model Priorscited by 1
  12. 2024Federated Progressive Self-Distillation with Logits Calibration for Personalized IIoT Edge Intelligencecited by 1
  13. 2023FedCliP: Federated Learning with Client Pruningcited by 0
  14. 2024KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgettingcited by 0
  15. 2024Federated Impression for Learning with Distributed Heterogeneous Datacited by 0
  16. 2025FlexFed: Mitigating Catastrophic Forgetting in Heterogeneous Federated Learning in Pervasive Computing Environmentscited by 0
  17. 2026FlashbackCL: Mitigating Temporal Forgetting in Federated Learningcited by 0