# GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2302.14307 · HTML: https://mlsanity.com/paper/2302.14307/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 48 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Asynchronous Federated Optimization](https://mlsanity.com/paper/1903.03934/) (2019)
- [Federated Learning for Internet of Things: A Comprehensive Survey](https://mlsanity.com/paper/2104.07914/) (2021)
- [Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning](https://mlsanity.com/paper/2101.11203/) (2021)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Federated Continual Learning with Weighted Inter-client Transfer](https://mlsanity.com/paper/2003.03196/) (2020)
- [Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning](https://mlsanity.com/paper/2008.03606/) (2020)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [Scalable and Order-robust Continual Learning with Additive Parameter Decomposition](https://mlsanity.com/paper/1902.09432/) (2019)
- [Overcoming Forgetting in Federated Learning on Non-IID Data](https://mlsanity.com/paper/1910.07796/) (2019)
- [FedCM: Federated Learning with Client-level Momentum](https://mlsanity.com/paper/2106.10874/) (2021)
- [Fast Federated Learning in the Presence of Arbitrary Device Unavailability](https://mlsanity.com/paper/2106.04159/) (2021)
- [HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images](https://mlsanity.com/paper/2112.10775/) (2021)

## Built on by (most cited first, 27 of 27)

- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Federated Continual Learning via Knowledge Fusion: A Survey](https://mlsanity.com/paper/2312.16475/) (2023)
- [Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity](https://mlsanity.com/paper/2405.16585/) (2024)
- [FwdLLM: Efficient FedLLM using Forward Gradient](https://mlsanity.com/paper/2308.13894/) (2023)
- [DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2309.13546/) (2023)
- [FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning](https://mlsanity.com/paper/2308.12532/) (2023)
- [FedProK: Trustworthy Federated Class-Incremental Learning via Prototypical Feature Knowledge Transfer](https://mlsanity.com/paper/2405.02685/) (2024)
- [Federated Class-Incremental Learning with Hierarchical Generative Prototypes](https://mlsanity.com/paper/2406.02447/) (2024)
- [Flashback: Understanding and Mitigating Forgetting in Federated Learning](https://mlsanity.com/paper/2402.05558/) (2024)
- [FedCFA: Alleviating Simpson's Paradox in Model Aggregation with Counterfactual Federated Learning](https://mlsanity.com/paper/2412.18904/) (2024)
- [pMixFed: Efficient Personalized Federated Learning through Adaptive Layer-Wise Mixup](https://mlsanity.com/paper/2501.11002/) (2025)
- [SNFinLLM: Systematic and Nuanced Financial Domain Adaptation of Chinese Large Language Models](https://mlsanity.com/paper/2408.02302/) (2024)
- [Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator](https://mlsanity.com/paper/2409.06955/) (2024)
- [DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learning](https://mlsanity.com/paper/2409.07734/) (2024)
- [Energy-Efficient and Real-Time Sensing for Federated Continual Learning via Sample-Driven Control](https://mlsanity.com/paper/2310.07497/) (2023)
- [Federated Cross-Training Learners for Robust Generalization under Data Heterogeneity](https://mlsanity.com/paper/2405.20046/) (2024)
- [TPFL: A Trustworthy Personalized Federated Learning Framework via Subjective Logic](https://mlsanity.com/paper/2410.12316/) (2024)
- [Addressing Label Shift in Distributed Learning via Entropy Regularization](https://mlsanity.com/paper/2502.02544/) (2025)
- [Personalized Recommendation Models in Federated Settings: A Survey](https://mlsanity.com/paper/2504.07101/) (2025)
- [Improving Generalization in Heterogeneous Federated Continual Learning via Spatio-Temporal Gradient Matching with Prototypical Coreset](https://mlsanity.com/paper/2506.12031/) (2025)
- [Federated Impression for Learning with Distributed Heterogeneous Data](https://mlsanity.com/paper/2409.07351/) (2024)
- [FlexFed: Mitigating Catastrophic Forgetting in Heterogeneous Federated Learning in Pervasive Computing Environments](https://mlsanity.com/paper/2505.13576/) (2025)
- [Intrinsic Training Signals for Federated Learning Aggregation](https://mlsanity.com/paper/2507.06813/) (2025)
- [DOLFIN: Balancing Stability and Plasticity in Federated Continual Learning](https://mlsanity.com/paper/2510.13567/) (2025)
- [Towards Robust Knowledge Removal in Federated Learning with High Data Heterogeneity](https://mlsanity.com/paper/2510.13606/) (2025)
- [FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgetting](https://mlsanity.com/paper/2608.15310/) (2026)

## Abstract (arXiv)

Federated Learning (FL) has emerged as a de facto machine learning area and received rapid increasing research interests from the community. However, catastrophic forgetting caused by data heterogeneity and partial participation poses distinctive challenges for FL, which are detrimental to the performance. To tackle the problems, we propose a new FL approach (namely GradMA), which takes inspiration from continual learning to simultaneously correct the server-side and worker-side update directions as well as take full advantage of server's rich computing and memory resources. Furthermore, we elaborate a memory reduction strategy to enable GradMA to accommodate FL with a large scale of workers. We then analyze convergence of GradMA theoretically under the smooth non-convex setting and show that its convergence rate achieves a linear speed up w.r.t the increasing number of sampled active workers. At last, our extensive experiments on various image classification tasks show that GradMA achieves significant performance gains in accuracy and communication efficiency compared to SOTA baselines.