mlsanityData to 14 Oct 2026

Faster Non-Convex Federated Learning via Global and Local Momentum

arXiv 2012.04061 ↗First posted 7 Dec 2020stat.ML cs.DC cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

We propose textttFedGLOMO, a novel federated learning (FL) algorithm with an iteration complexity of O(ε-1.5) to converge to an ε-stationary point (i.e., mathbbE[|nabla f(bmx)|2] ≤ ε) for smooth non-convex functions -- under arbitrary client heterogeneity and compressed communication -- compared to the O(ε-2) complexity of most prior works. Our key algorithmic idea that enables achieving this improved complexity is based on the observation that the convergence in FL is hampered by two sources of high variance: (i) the global server aggregation step with multiple local updates, exacerbated by client heterogeneity, and (ii) the noise of the local client-level stochastic gradients. By modeling the server aggregation step as a generalized gradient-type update, we propose a variance-reducing momentum-based global update at the server, which when applied in conjunction with variance-reduced local updates at the clients, enables textttFedGLOMO to enjoy an improved convergence rate. Moreover, we derive our results under a novel and more realistic client-heterogeneity assumption which we verify empirically -- unlike prior assumptions that are hard to verify. Our experiments illustrate the intrinsic variance reduction effect of textttFedGLOMO, which implicitly suppresses client-drift in heterogeneous data distribution settings and promotes communication efficiency.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 10; 2023: 14; 2024: 12; 2025: 4; 2026: 2.

48 papers.

Sort:

By area:

  1. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  2. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  3. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 64
  4. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  5. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  6. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 38
  7. 2022FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reductioncited by 35
  8. 2022Faster Adaptive Federated Learningcited by 35
  9. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 31
  10. 2022Communication-Efficient Federated Learning with Accelerated Client Gradientcited by 24
  11. 2021Towards Federated Learning on Time-Evolving Heterogeneous Datacited by 23
  12. 2021Robust Training in High Dimensions via Block Coordinate Geometric Median Descentcited by 20
36 more
  1. 2022Privacy-Aware Compression for Federated Data Analysiscited by 17
  2. 2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
  3. 2022CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimizationcited by 11
  4. 2022Online Federated Learning via Non-Stationary Detection and Adaptation amidst Concept Driftcited by 10
  5. 2023Over-the-Air Federated Learning and Optimizationcited by 10
  6. 2023A Communication-Efficient Adaptive Algorithm for Federated Learning under Cumulative Regretcited by 7
  7. 2025Federated Learning for Diffusion Modelscited by 7
  8. 2021MDPGT: Momentum-based Decentralized Policy Gradient Trackingcited by 6
  9. 2024FedLion: Faster Adaptive Federated Optimization with Fewer Communicationcited by 6
  10. 2024Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptionscited by 6
  11. 2022Adaptive Federated Minimax Optimization with Lower Complexitiescited by 5
  12. 2024Towards Faster Decentralized Stochastic Optimization with Communication Compressioncited by 5
  13. 2026Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variancecited by 5
  14. 2022Federated Learning with a Sampling Algorithm under Isoperimetrycited by 4
  15. 2023TAMUNA: Doubly Accelerated Distributed Optimization under Partial Participationcited by 4
  16. 2023Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noisecited by 4
  17. 2023Federated Conditional Stochastic Optimizationcited by 4
  18. 2024A New Theoretical Perspective on Data Heterogeneity in Federated Optimizationcited by 4
  19. 2023FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learningcited by 3
  20. 2023Federated Learning: A Cutting-Edge Survey of the Latest Advancements and Applicationscited by 3
  21. 2023Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentumcited by 3
  22. 2022Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learningcited by 2
  23. 2023FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averagingcited by 2
  24. 2024FedECADO: A Dynamical System Model of Federated Learningcited by 2
  25. 2024Non-Convex Optimization in Federated Learning via Variance Reduction and Adaptive Learningcited by 2
  26. 2025GC-Fed: Gradient Centralized Federated Learning with Partial Client Participationcited by 2
  27. 2024Localized Distributional Robustness in Submodular Multi-Task Subset Selectioncited by 1
  28. 2024Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasetscited by 1
  29. 2024Submodular Maximization Approaches for Equitable Client Selection in Federated Learningcited by 1
  30. 2026RAMPAGE: RAndomized Mid-Point for debiAsed Gradient Extrapolationcited by 1
  31. 2023Communication-Efficient Zeroth-Order Distributed Online Optimization: Algorithm, Theory, and Applicationscited by 0
  32. 2024FedEGG: Federated Learning with Explicit Global Guidancecited by 0
  33. 2024Convergent Differential Privacy Analysis for General Federated Learningcited by 0
  34. 2024Equitable Federated Learning with Activation Clusteringcited by 0
  35. 2025Robust Federated Learning in Unreliable Wireless Networks: A Client Selection Approachcited by 0
  36. 2025Adaptive Federated Learning via Dynamical System Modelcited by 0