mlsanityData to 7 Oct 2026

Momentum Approximation in Asynchronous Private Federated Learning

arXiv 2402.09247 ↗First posted 14 Feb 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Asynchronous protocols have been shown to improve the scalability of federated learning (FL) with a massive number of clients. Meanwhile, momentum-based methods can achieve the best model quality in synchronous FL. However, naively applying momentum in asynchronous FL algorithms leads to slower convergence and degraded model performance. It is still unclear how to effective combinie these two techniques together to achieve a win-win. In this paper, we find that asynchrony introduces implicit bias to momentum updates. In order to address this problem, we propose momentum approximation that minimizes the bias by finding an optimal weighted average of all historical model updates. Momentum approximation is compatible with secure aggregation as well as differential privacy, and can be easily integrated in production FL systems with a minor communication and storage cost. We empirically demonstrate that on benchmark FL datasets, momentum approximation can achieve 1.15 textrm--4× speed up in convergence compared to naively combining asynchronous FL with momentum.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 51 of its 68 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

51 of 68 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2020Adaptive Federated Optimizationcited by 1,036
  7. 2019Deep Leakage from Gradientscited by 1,004
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  9. 2019Towards Federated Learning at Scale: System Designcited by 826
  10. 2017Renyi Differential Privacycited by 643
  11. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
  12. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
38 more
  1. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  2. 2021A Field Guide to Federated Optimizationcited by 312
  3. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 269
  4. 2019Asynchronous Federated Optimizationcited by 264
  5. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  6. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  7. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  8. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  9. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  10. 2022On-Device Training Under 256KB Memorycited by 158
  11. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  12. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  13. 2019Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregationcited by 99
  14. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
  15. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 89
  16. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 84
  17. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  18. 2018Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGDcited by 68
  19. 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
  20. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  21. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
  22. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  23. 2015Staleness-aware Async-SGD for Distributed Deep Learningcited by 43
  24. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  25. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  26. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  27. 2020FedADC: Accelerated Federated Learning with Drift Controlcited by 26
  28. 2022FLAIR: Federated Learning Annotated Image Repositorycited by 22
  29. 2020Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noisecited by 21
  30. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  31. 2023FLINT: A Platform for Federated Learning Integrationcited by 17
  32. 2017YellowFin and the Art of Momentum Tuningcited by 16
  33. 2022Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devicescited by 16
  34. 2023On the Role of Server Momentum in Federated Learningcited by 16
  35. 2019Gap Aware Mitigation of Gradient Stalenesscited by 14
  36. 2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
  37. 2020Pipelined Backpropagation at Scale: Training Large Models without Batchescited by 11
  38. 2019Taming Momentum in a Distributed Asynchronous Environmentcited by 9

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  2. 2025Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learningcited by 0
  3. 2026FedPSA: Modeling Behavioral Staleness in Asynchronous Federated Learningcited by 0