# Faster Non-Convex Federated Learning via Global and Local Momentum

arXiv: https://arxiv.org/abs/2012.04061 · HTML: https://mlsanity.com/paper/2012.04061/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2020.

## Built on by (most cited first, 30 of 48)

- [Stochastic Controlled Averaging for Federated Learning with Communication Compression](https://mlsanity.com/paper/2308.08165/) (2023)
- [Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data](https://mlsanity.com/paper/2102.04761/) (2021)
- [FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learning](https://mlsanity.com/paper/2207.14130/) (2022)
- [Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape](https://mlsanity.com/paper/2305.11584/) (2023)
- [Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning](https://mlsanity.com/paper/2312.17493/) (2023)
- [Momentum Benefits Non-IID Federated Learning Simply and Provably](https://mlsanity.com/paper/2306.16504/) (2023)
- [FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reduction](https://mlsanity.com/paper/2205.13462/) (2022)
- [Faster Adaptive Federated Learning](https://mlsanity.com/paper/2212.00974/) (2022)
- [Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning](https://mlsanity.com/paper/2102.03198/) (2021)
- [Communication-Efficient Federated Learning with Accelerated Client Gradient](https://mlsanity.com/paper/2201.03172/) (2022)
- [Towards Federated Learning on Time-Evolving Heterogeneous Data](https://mlsanity.com/paper/2112.13246/) (2021)
- [Robust Training in High Dimensions via Block Coordinate Geometric Median Descent](https://mlsanity.com/paper/2106.08882/) (2021)
- [Privacy-Aware Compression for Federated Data Analysis](https://mlsanity.com/paper/2203.08134/) (2022)
- [Faster Rates for Compressed Federated Learning with Client-Variance Reduction](https://mlsanity.com/paper/2112.13097/) (2021)
- [CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimization](https://mlsanity.com/paper/2210.13277/) (2022)
- [Online Federated Learning via Non-Stationary Detection and Adaptation amidst Concept Drift](https://mlsanity.com/paper/2211.12578/) (2022)
- [Over-the-Air Federated Learning and Optimization](https://mlsanity.com/paper/2310.10089/) (2023)
- [A Communication-Efficient Adaptive Algorithm for Federated Learning under Cumulative Regret](https://mlsanity.com/paper/2301.08869/) (2023)
- [Federated Learning for Diffusion Models](https://mlsanity.com/paper/2503.06426/) (2025)
- [MDPGT: Momentum-based Decentralized Policy Gradient Tracking](https://mlsanity.com/paper/2112.02813/) (2021)
- [FedLion: Faster Adaptive Federated Optimization with Fewer Communication](https://mlsanity.com/paper/2402.09941/) (2024)
- [Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptions](https://mlsanity.com/paper/2406.01959/) (2024)
- [Adaptive Federated Minimax Optimization with Lower Complexities](https://mlsanity.com/paper/2211.07303/) (2022)
- [Towards Faster Decentralized Stochastic Optimization with Communication Compression](https://mlsanity.com/paper/2405.20114/) (2024)
- [Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance](https://mlsanity.com/paper/2604.16620/) (2026)
- [Federated Learning with a Sampling Algorithm under Isoperimetry](https://mlsanity.com/paper/2206.00920/) (2022)
- [TAMUNA: Doubly Accelerated Distributed Optimization under Partial Participation](https://mlsanity.com/paper/2302.09832/) (2023)
- [Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise](https://mlsanity.com/paper/2307.07406/) (2023)
- [Federated Conditional Stochastic Optimization](https://mlsanity.com/paper/2310.02524/) (2023)
- [A New Theoretical Perspective on Data Heterogeneity in Federated Optimization](https://mlsanity.com/paper/2407.15567/) (2024)

## Abstract (arXiv)

We propose textttFedGLOMO, a novel federated learning (FL) algorithm with an iteration complexity of O(ε-1.5) to converge to an ε-stationary point (i.e., mathbbE[|nabla f(bmx)|2] ≤ ε) for smooth non-convex functions -- under arbitrary client heterogeneity and compressed communication -- compared to the O(ε-2) complexity of most prior works. Our key algorithmic idea that enables achieving this improved complexity is based on the observation that the convergence in FL is hampered by two sources of high variance: (i) the global server aggregation step with multiple local updates, exacerbated by client heterogeneity, and (ii) the noise of the local client-level stochastic gradients. By modeling the server aggregation step as a generalized gradient-type update, we propose a variance-reducing momentum-based global update at the server, which when applied in conjunction with variance-reduced local updates at the clients, enables textttFedGLOMO to enjoy an improved convergence rate. Moreover, we derive our results under a novel and more realistic client-heterogeneity assumption which we verify empirically -- unlike prior assumptions that are hard to verify. Our experiments illustrate the intrinsic variance reduction effect of textttFedGLOMO, which implicitly suppresses client-drift in heterogeneous data distribution settings and promotes communication efficiency.