mlsanityData to 14 Oct 2026

On the Role of Server Momentum in Federated Learning

arXiv 2312.12670 ↗First posted 19 Dec 2023cs.LG cs.AI cs.DCcited by 16

Not on the map. Why?

Abstract (arXiv)

Federated Averaging (FedAvg) is known to experience convergence issues when encountering significant clients system heterogeneity and data heterogeneity. Server momentum has been proposed as an effective mitigation. However, existing server momentum works are restrictive in the momentum formulation, do not properly schedule hyperparameters and focus only on system homogeneous settings, which leaves the role of server momentum still an under-explored problem. In this paper, we propose a general framework for server momentum, that (a) covers a large class of momentum schemes that are unexplored in federated learning (FL), (b) enables a popular stagewise hyperparameter scheduler, (c) allows heterogeneous and asynchronous local computing. We provide rigorous convergence analysis for the proposed framework. To our best knowledge, this is the first work that thoroughly analyzes the performances of server momentum with a hyperparameter scheduler and system heterogeneity. Extensive experiments validate the effectiveness of our proposed framework.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 48 of its 70 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

48 of 70 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2017Inductive Representation Learning on Large Graphscited by 6,320
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  11. 2020Adaptive Federated Optimizationcited by 1,046
  12. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
36 more
  1. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2016Wide & Deep Learning for Recommender Systemscited by 747
  4. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  5. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  6. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  7. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 352
  8. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  9. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  10. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  11. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  12. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  13. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  14. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  15. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  16. 2020Momentum Improves Normalized SGDcited by 139
  17. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  18. 2020Optimal Client Sampling for Federated Learningcited by 124
  19. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 119
  20. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 116
  21. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  22. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  23. 2014Deep learning with Elastic Averaging SGDcited by 90
  24. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  25. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  26. 2019Active Federated Learningcited by 63
  27. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 49
  28. 2018Quasi-hyperbolic momentum and Adam for deep learningcited by 48
  29. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  30. 2020Communication-Efficient Federated Learning via Optimal Client Samplingcited by 39
  31. 2020Fast OSCAR and OWL Regression via Safe Screening Rulescited by 25
  32. 2023Adaptive Test-Time Personalization for Federated Learningcited by 23
  33. 2021On the Convergence of Step Decay Step-Size for Stochastic Optimizationcited by 21
  34. 2020Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availabilitycited by 20
  35. 2023Solving a Class of Non-Convex Minimax Optimization in Federated Learningcited by 10
  36. 2023Enhance Diffusion to Improve Robust Generalizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyondcited by 25
  2. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 9
  3. 2024Ordered Momentum for Asynchronous SGDcited by 5
  4. 2025DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Modelscited by 4
  5. 2024Momentum Approximation in Asynchronous Private Federated Learningcited by 3
  6. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  7. 2024Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learningcited by 2
  8. 2024Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex functionscited by 1
  9. 2025FedWCM: Unleashing the Potential of Momentum-based Federated Learning in Long-Tailed Scenarioscited by 1
  10. 2024Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awarenesscited by 0
  11. 2024Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimizationcited by 0
  12. 2025Anomaly Detection in Electric Vehicle Charging Stations Using Federated Learningcited by 0
4 more
  1. 2025Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optimacited by 0
  2. 2026LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communicationcited by 0
  3. 2026A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimizationcited by 0
  4. 2026FedCMAPSS: A Benchmark for Federated Learning in Remaining Useful Life Estimationcited by 0