mlsanityData to 7 Oct 2026

Is Local SGD Better than Minibatch SGD?

arXiv 2002.07839 ↗First posted 18 Feb 2020cs.LG math.OC stat.MLcited by 151

Not on the map. Why?

Abstract (arXiv)

We study local SGD (also known as parallel SGD and federated averaging), a natural and frequently used stochastic distributed optimization method. Its theoretical foundations are currently lacking and we highlight how all existing error guarantees in the convex setting are dominated by a simple baseline, minibatch SGD. (1) For quadratic objectives we prove that local SGD strictly dominates minibatch SGD and that accelerated local SGD is minimax optimal for quadratics; (2) For general convex objectives we provide the first guarantee that at least sometimes improves over minibatch SGD; (3) We show that indeed local SGD does not dominate minibatch SGD by presenting a lower bound on the performance of local SGD that is worse than the minibatch SGD guarantee.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 40; 2022: 31; 2023: 30; 2024: 25; 2025: 13; 2026: 12.

151 papers.

Sort:

  1. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  2. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  3. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  4. 2021Distributionally Robust Federated Averagingcited by 104
  5. 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
  6. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  7. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
  8. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  9. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  10. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  11. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  12. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
38 more
  1. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  2. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  3. 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
  4. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  5. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  6. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 36
  7. 2022Faster Adaptive Federated Learningcited by 35
  8. 2021Asynchronous Stochastic Optimization Robust to Arbitrary Delayscited by 32
  9. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  10. 2021The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communicationcited by 30
  11. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 30
  12. 2021Private Federated Learning Without a Trusted Server: Optimal Algorithms for Convex Lossescited by 30
  13. 2021FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimizationcited by 29
  14. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  15. 20223PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregationcited by 27
  16. 2022Private Non-Convex Federated Learning Without a Trusted Servercited by 26
  17. 2021Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfercited by 23
  18. 2021Towards Federated Learning on Time-Evolving Heterogeneous Datacited by 23
  19. 2022Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learningcited by 23
  20. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  21. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  22. 2021Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexitycited by 21
  23. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  24. 2021Efficient Algorithms for Federated Saddle Point Optimizationcited by 20
  25. 2021Exploiting Heterogeneity in Robust Federated Best-Arm Identificationcited by 20
  26. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  27. 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
  28. 2021FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learningcited by 19
  29. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
  30. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
  31. 2022Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Proxcited by 19
  32. 2022Federated Optimization Algorithms with Random Reshuffling and Gradient Compressioncited by 18
  33. 2023Achieving Linear Speedup in Non-IID Federated Bilevel Learningcited by 18
  34. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  35. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 14
  36. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  37. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Trainingcited by 13
  38. 2021Compositional federated learning: Applications in distributionally robust averaging and meta learningcited by 12

50 most cited of 151 All 151 (JSON)