mlsanityData to 7 Oct 2026

Local SGD: Unified Theory and New Efficient Methods

arXiv 2011.02828 ↗First posted 3 Nov 2020cs.LGcited by 88

Not on the map. Why?

Abstract (arXiv)

We present a unified framework for analyzing local SGD methods in the convex and strongly convex regimes for distributed/federated training of supervised machine learning models. We recover several known methods as a special case of our general framework, including Local-SGD/FedAvg, SCAFFOLD, and several variants of SGD not originally designed for federated learning. Our framework covers both the identical and heterogeneous data settings, supports both random and deterministic number of local steps, and can work with a wide array of local stochastic gradient estimators, including shifted estimators which are able to adjust the fixed points of local iterations for faster convergence. As an application of our framework, we develop multiple novel FL optimizers which are superior to existing methods. In particular, we develop the first linearly converging local SGD method which does not require any data homogeneity or other strong assumptions.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 28; 2023: 15; 2024: 12; 2025: 10; 2026: 7.

88 papers.

Sort:

  1. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  2. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  3. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  4. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  5. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  6. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
  7. 2022BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compressioncited by 41
  8. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  9. 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
  10. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  11. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  12. 2021FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimizationcited by 29
38 more
  1. 2021Decentralized Local Stochastic Extra-Gradient for Variational Inequalitiescited by 25
  2. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 25
  3. 2022Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimizationcited by 23
  4. 2022Can 5th Generation Local Training Methods Support Client Sampling? Yes!cited by 22
  5. 2022On the Performance of Gradient Tracking with Local Updatescited by 21
  6. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  7. 2021Efficient Algorithms for Federated Saddle Point Optimizationcited by 20
  8. 2021Exploiting Heterogeneity in Robust Federated Best-Arm Identificationcited by 20
  9. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  10. 2021FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learningcited by 19
  11. 2022Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Proxcited by 19
  12. 2022Federated Optimization Algorithms with Random Reshuffling and Gradient Compressioncited by 18
  13. 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 17
  14. 2023AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithmscited by 16
  15. 2021Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learningcited by 14
  16. 2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
  17. 2022Coresets for Vertical Federated Learning: Regularized Linear Regression and K-Means Clusteringcited by 14
  18. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 14
  19. 2021A Non-parametric View of FedAvg and FedProx: Beyond Stationary Pointscited by 13
  20. 2022Federated Learning with Flexible Controlcited by 12
  21. 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 12
  22. 2021Secure Distributed Training at Scalecited by 11
  23. 2022Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregationcited by 11
  24. 2022CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimizationcited by 11
  25. 2021FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learningcited by 10
  26. 2022Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topologycited by 10
  27. 2023Over-the-Air Federated Learning and Optimizationcited by 10
  28. 2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
  29. 2023Improving Accelerated Federated Learning with Compression and Importance Samplingcited by 7
  30. 2026FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learningcited by 7
  31. 2022Federated Learning on Adaptively Weighted Nodes by Bilevel Optimizationcited by 6
  32. 2022Personalized Federated Learning with Communication Compressioncited by 6
  33. 2023Distributed Personalized Empirical Risk Minimizationcited by 6
  34. 2024SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learningcited by 6
  35. 2023Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learningcited by 5
  36. 2024FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Modelscited by 5
  37. 2025Multiplayer Federated Learning: Reaching Equilibrium with Less Communicationcited by 5
  38. 2022Faster Convergence of Local SGD for Over-Parameterized Modelscited by 4

50 most cited of 88 All 88 (JSON)