mlsanityData to 14 Oct 2026

Momentum Benefits Non-IID Federated Learning Simply and Provably

arXiv 2306.16504 ↗First posted 28 Jun 2023cs.LG math.OCcited by 38

Not on the map. Why?

Abstract (arXiv)

Federated learning is a powerful paradigm for large-scale machine learning, but it faces significant challenges due to unreliable network connections, slow communication, and substantial data heterogeneity across clients. FedAvg and SCAFFOLD are two prominent algorithms to address these challenges. In particular, FedAvg employs multiple local updates before communicating with a central server, while SCAFFOLD maintains a control variable on each client to compensate for ``client drift'' in its local updates. Various methods have been proposed to enhance the convergence of these two algorithms, but they either make impractical adjustments to the algorithmic structure or rely on the assumption of bounded data heterogeneity. This paper explores the utilization of momentum to enhance the performance of FedAvg and SCAFFOLD. When all clients participate in the training process, we demonstrate that incorporating momentum allows FedAvg to converge without relying on the assumption of bounded data heterogeneity even using a constant local learning rate. This is novel and fairly surprising as existing analyses for FedAvg require bounded data heterogeneity even with diminishing local learning rates. In partial client participation, we show that momentum enables SCAFFOLD to converge provably faster without imposing any additional assumptions. Furthermore, we use momentum to develop new variance-reduced extensions of FedAvg and SCAFFOLD, which exhibit state-of-the-art convergence rates. Our experimental results support all theoretical findings.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 57 of its 71 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

57 of 71 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2020Adaptive Federated Optimizationcited by 1,046
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  10. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  11. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  12. 2021Federated Learning Based on Dynamic Regularizationcited by 562
38 more
  1. 2021Towards Personalized Federated Learningcited by 464
  2. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  3. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  4. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  5. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  6. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  7. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  8. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  9. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  10. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  11. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  12. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  13. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  14. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  15. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  16. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  17. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 119
  18. 2022Federated Learning with Partial Model Personalizationcited by 114
  19. 2016NEXT: In-Network Nonconvex Optimizationcited by 107
  20. 2018Distributed Stochastic Gradient Tracking Methodscited by 106
  21. 2021Asynchronous Federated Learning on Heterogeneous Devices: A Surveycited by 93
  22. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 92
  23. 2020VAFL: a Method of Vertical Asynchronous Federated Learningcited by 87
  24. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  25. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  26. 2018A Unified Analysis of Stochastic Momentum Methods for Deep Learningcited by 55
  27. 2020An improved convergence analysis for decentralized online stochastic non-convex optimizationcited by 53
  28. 2017Exact Diffusion for Distributed Optimization and Learning --- Part I: Algorithm Developmentcited by 50
  29. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  30. 2020Faster Non-Convex Federated Learning via Global and Local Momentumcited by 48
  31. 2021A Unified and Refined Convergence Analysis for Non-Convex Decentralized Learningcited by 44
  32. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  33. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  34. 2019On the Influence of Bias-Correction on Distributed Stochastic Optimizationcited by 34
  35. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 31
  36. 2023Momentum Provably Improves Error Feedback!cited by 31
  37. 2023Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditionscited by 31
  38. 2021Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGDcited by 29

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 17; 2026: 10.

38 papers.

Sort:

By area:

  1. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  2. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 8
  3. 2024Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysiscited by 8
  4. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 8
  5. 2025On Provable Benefits of Muon in Federated Learningcited by 7
  6. 2024Towards Faster Decentralized Stochastic Optimization with Communication Compressioncited by 5
  7. 2024Asynchronous Heterogeneous Linear Quadratic Regulator Designcited by 4
  8. 2024Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applicationscited by 4
  9. 2025FedCanon: Non-Convex Composite Federated Learning with Efficient Proximal Operation on Heterogeneous Datacited by 4
  10. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
  11. 2023Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentumcited by 3
  12. 2024Review of Mathematical Optimization in Federated Learningcited by 3
26 more
  1. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  2. 2025Gradient Correction in Federated Learning with Adaptive Optimizationcited by 2
  3. 2026Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignmentcited by 2
  4. 2023Achieving Linear Speedup with ProxSkip in Distributed Stochastic Optimizationcited by 1
  5. 2025DBA-DFL: Towards Distributed Backdoor Attacks with Network Detection in Decentralized Federated Learningcited by 1
  6. 2025Optimized Local Updates in Federated Learning via Reinforcement Learningcited by 1
  7. 2025FedWCM: Unleashing the Potential of Momentum-based Federated Learning in Long-Tailed Scenarioscited by 1
  8. 2025A Byzantine-Robust Federated Learning with Nesterov Momentumcited by 1
  9. 2025pFedBBN: A Personalized Federated Test-Time Adaptation with Balanced Batch Normalization for Class-Imbalanced Datacited by 1
  10. 2026FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projectioncited by 1
  11. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  12. 2024FedEGG: Federated Learning with Explicit Global Guidancecited by 0
  13. 2024Accelerating Distributed Optimization: A Primal-Dual Perspective on Local Stepscited by 0
  14. 2025FedOSAA: Improving Federated Learning with One-Step Anderson Accelerationcited by 0
  15. 2025Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentumcited by 0
  16. 2025FedCLAM: Client Adaptive Momentum with Foreground Intensity Matching for Federated Medical Image Segmentationcited by 0
  17. 2025Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optimacited by 0
  18. 2025On the Limits of Momentum in Decentralized and Federated Optimizationcited by 0
  19. 2025Communication-Efficient Neural Tangent Kernels for Heterogeneous Decentralized Federated Learningcited by 0
  20. 2026Byzantine-Robust Distributed SGD: A Unified Analysis and Tight Error Boundscited by 0
  21. 2026FedVSSAM: Mitigating Flatness Incompatibility in Sharpness-Aware Federated Learningcited by 0
  22. 2026Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learningcited by 0
  23. 2026Federated Compositional Muon Optimizer for Matrix-Wise Modelscited by 0
  24. 2026A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimizationcited by 0
  25. 2026Momentum-Guided Federated Split Distillation for Personalized Temporal Edge Intelligencecited by 0
  26. 2026CTP-FL: Common-Trajectory Gradient Prediction for Federated Learningcited by 0