mlsanityData to 14 Oct 2026

Flashback: Understanding and Mitigating Forgetting in Federated Learning

arXiv 2402.05558 ↗First posted 8 Feb 2024cs.LG cs.AI cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

In Federated Learning (FL), forgetting, or the loss of knowledge across rounds, hampers algorithm convergence, particularly in the presence of severe data heterogeneity among clients. This study explores the nuances of this issue, emphasizing the critical role of forgetting in FL's inefficient learning within heterogeneous data contexts. Knowledge loss occurs in both client-local updates and server-side aggregation steps; addressing one without the other fails to mitigate forgetting. We introduce a metric to measure forgetting granularly, ensuring distinct recognition amid new knowledge acquisition. Leveraging these insights, we propose Flashback, an FL algorithm with a dynamic distillation approach that is used to regularize the local models, and effectively aggregate their knowledge. Across different benchmarks, Flashback outperforms other methods, mitigates forgetting, and achieves faster round-to-target-accuracy, by converging in 6 to 16 rounds.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

26 of 41 references.

References · 26

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2019Towards Federated Learning at Scale: System Designcited by 832
  10. 2018LEAF: A Benchmark for Federated Settingscited by 792
  11. 2021Model-Contrastive Federated Learningcited by 766
  12. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
14 more
  1. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  2. 2020Federated Learning with Matched Averagingcited by 564
  3. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  4. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 204
  5. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 195
  6. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  7. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  8. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  9. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  10. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  11. 2021Resource-Efficient Federated Learningcited by 26
  12. 2021FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systemscited by 16
  13. 2023FilFL: Client Filtering for Optimized Client Participation in Federated Learningcited by 5
  14. 2017Stochastic, Distributed and Federated Optimization for Machine Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Where is the Testbed for my Federated Learning Research?cited by 0
  2. 2025FlexFed: Mitigating Catastrophic Forgetting in Heterogeneous Federated Learning in Pervasive Computing Environmentscited by 0
  3. 2026FlashbackCL: Mitigating Temporal Forgetting in Federated Learningcited by 0
  4. 2026FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgettingcited by 0