mlsanityData to 7 Oct 2026

On the Limits of Momentum in Decentralized and Federated Optimization

arXiv 2511.20168 ↗First posted 25 Nov 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent works have explored the use of momentum in local methods to enhance distributed SGD. This is particularly appealing in Federated Learning (FL), where momentum intuitively appears as a solution to mitigate the effects of statistical heterogeneity. Despite recent progress in this direction, it is still unclear if momentum can guarantee convergence under unbounded heterogeneity in decentralized scenarios, where only some workers participate at each round. In this work we analyze momentum under cyclic client participation, and theoretically prove that it remains inevitably affected by statistical heterogeneity. Similarly to SGD, we prove that decreasing step-sizes do not help either: in fact, any schedule decreasing faster than Thetaleft(1/tright) leads to convergence to a constant value that depends on the initialization and the heterogeneity bound. Numerical results corroborate the theory, and deep learning experiments confirm its relevance for realistic settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 24 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

22 of 24 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2018Group Normalizationcited by 1,514
  4. 2020Adaptive Federated Optimizationcited by 1,036
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  6. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  7. 2020Federated Visual Classification with Real-World Data Distributioncited by 126
  8. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  9. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 117
  10. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  11. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  12. 2019How Good is SGD with Random Shuffling?cited by 63
10 more
  1. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  2. 2020FedADC: Accelerated Federated Learning with Drift Controlcited by 26
  3. 2022Communication-Efficient Federated Learning with Accelerated Client Gradientcited by 24
  4. 2018Tight Dimension Independent Lower Bound on the Expected Convergence Rate for Diminishing Step Sizes in SGDcited by 18
  5. 2018Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning ratescited by 11
  6. 2023Enhance Local Consistency in Federated Learning: A Multi-Step Inertial Momentum Approachcited by 11
  7. 2024On the Last-Iterate Convergence of Shuffling Gradient Methodscited by 9
  8. 2023On Convergence of Incremental Gradient for Non-Convex Smooth Functionscited by 7
  9. 2023Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentumcited by 3
  10. 2025Incremental Gradient Descent with Small Epoch Counts is Surprisingly Slow on Ill-Conditioned Problemscited by 1

Built on by

Not cited yet.