# Acceleration of Federated Learning with Alleviated Forgetting in Local Training

arXiv: https://arxiv.org/abs/2203.02645 · HTML: https://mlsanity.com/paper/2203.02645/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 5 Mar 2022.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2106.05001/) (2021)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [FetchSGD: Communication-Efficient Federated Learning with Sketching](https://mlsanity.com/paper/2007.07682/) (2020)
- [Federated Visual Classification with Real-World Data Distribution](https://mlsanity.com/paper/2003.08082/) (2020)
- [Overcoming Forgetting in Federated Learning on Non-IID Data](https://mlsanity.com/paper/1910.07796/) (2019)
- [Federated Accelerated Stochastic Gradient Descent](https://mlsanity.com/paper/2006.08950/) (2020)

## Built on by (most cited first, 29 of 29)

- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Learning Federated Visual Prompt in Null Space for MRI Reconstruction](https://mlsanity.com/paper/2303.16181/) (2023)
- [Personalized Federated Learning under Mixture of Distributions](https://mlsanity.com/paper/2305.01068/) (2023)
- [Relaxed Contrastive Learning for Federated Learning](https://mlsanity.com/paper/2401.04928/) (2024)
- [GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting](https://mlsanity.com/paper/2302.14307/) (2023)
- [Meta Knowledge Condensation for Federated Learning](https://mlsanity.com/paper/2209.14851/) (2022)
- [FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation](https://mlsanity.com/paper/2205.14900/) (2022)
- [Accelerated Federated Learning with Decoupled Adaptive Optimization](https://mlsanity.com/paper/2207.07223/) (2022)
- [SphereFed: Hyperspherical Federated Learning](https://mlsanity.com/paper/2207.09413/) (2022)
- [FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning](https://mlsanity.com/paper/2308.12532/) (2023)
- [Feature Matching Data Synthesis for Non-IID Federated Learning](https://mlsanity.com/paper/2308.04761/) (2023)
- [Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning](https://mlsanity.com/paper/2304.05260/) (2023)
- [Towards Instance-adaptive Inference for Federated Learning](https://mlsanity.com/paper/2308.06051/) (2023)
- [Fake It Till Make It: Federated Learning with Consensus-Oriented Generation](https://mlsanity.com/paper/2312.05966/) (2023)
- [FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning](https://mlsanity.com/paper/2311.16584/) (2023)
- [Flashback: Understanding and Mitigating Forgetting in Federated Learning](https://mlsanity.com/paper/2402.05558/) (2024)
- [FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning](https://mlsanity.com/paper/2404.09210/) (2024)
- [Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data](https://mlsanity.com/paper/2504.14694/) (2025)
- [An Element-Wise Weights Aggregation Method for Federated Learning](https://mlsanity.com/paper/2404.15919/) (2024)
- [pMixFed: Efficient Personalized Federated Learning through Adaptive Layer-Wise Mixup](https://mlsanity.com/paper/2501.11002/) (2025)
- [FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularization](https://mlsanity.com/paper/2209.10537/) (2022)
- [Data Heterogeneity and Forgotten Labels in Split Federated Learning](https://mlsanity.com/paper/2511.09736/) (2025)
- [Structured Cooperative Learning with Graphical Model Priors](https://mlsanity.com/paper/2306.09595/) (2023)
- [Federated Progressive Self-Distillation with Logits Calibration for Personalized IIoT Edge Intelligence](https://mlsanity.com/paper/2412.00410/) (2024)
- [FedCliP: Federated Learning with Client Pruning](https://mlsanity.com/paper/2301.06768/) (2023)
- [KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgetting](https://mlsanity.com/paper/2404.12846/) (2024)
- [Federated Impression for Learning with Distributed Heterogeneous Data](https://mlsanity.com/paper/2409.07351/) (2024)
- [FlexFed: Mitigating Catastrophic Forgetting in Heterogeneous Federated Learning in Pervasive Computing Environments](https://mlsanity.com/paper/2505.13576/) (2025)
- [FlashbackCL: Mitigating Temporal Forgetting in Federated Learning](https://mlsanity.com/paper/2606.03939/) (2026)

## Abstract (arXiv)

Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy by independently training local models on each client and then aggregating parameters on a central server, thereby producing an effective global model. Although a variety of FL algorithms have been proposed, their training efficiency remains low when the data are not independently and identically distributed (non-i.i.d.) across different clients. We observe that the slow convergence rates of the existing methods are (at least partially) caused by the catastrophic forgetting issue during the local training stage on each individual client, which leads to a large increase in the loss function concerning the previous training data at the other clients. Here, we propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage by regularizing locally trained parameters with the loss on generated pseudo data, which encode the knowledge of previous training data learned by the global model. Our comprehensive experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep and the clients' data are extremely non-i.i.d., but is also able to protect privacy better in classification problems and more robust against gradient inversion attacks. The code is available at: https://github.com/Zoesgithub/FedReg.