mlsanityData to 7 Oct 2026

Don't Use Large Mini-Batches, Use Local SGD

arXiv 1808.07217 ↗First posted 22 Aug 2018cs.LG stat.MLcited by 201

Not on the map. Why?

Abstract (arXiv)

Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a post-local SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of local SGD variants.

Key facts. Cited by 201 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 61; 2022: 32; 2023: 38; 2024: 39; 2025: 21; 2026: 9.

201 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2021Personalized Federated Learning using Hypernetworkscited by 210
  3. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  4. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  5. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  6. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
  7. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  8. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  9. 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
  10. 2021Personalized Federated Learning with Gaussian Processescited by 76
  11. 2021On Large-Cohort Training for Federated Learningcited by 73
  12. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
38 more
  1. 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
  2. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  3. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  4. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  5. 2021Consensus Control for Decentralized Deep Learningcited by 49
  6. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  7. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
  8. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
  9. 2022Federated Learning via Inexact ADMMcited by 44
  10. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  11. 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
  12. 2021Optimality and Stability in Federated Learning: A Game-theoretic Approachcited by 40
  13. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  14. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  15. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  16. 2024Hierarchical Split Federated Learning: Convergence Analysis and System Optimizationcited by 38
  17. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  18. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  19. 2022Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Datacited by 36
  20. 2022FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reductioncited by 35
  21. 2023Decentralized Gradient Tracking with Local Stepscited by 32
  22. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 30
  23. 2021SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networkscited by 30
  24. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  25. 2021FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimizationcited by 29
  26. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  27. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  28. 2021Accelerating Gossip SGD with Periodic Global Averagingcited by 25
  29. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  30. 2021Towards Federated Learning on Time-Evolving Heterogeneous Datacited by 23
  31. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  32. 2021Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexitycited by 21
  33. 2022On the Performance of Gradient Tracking with Local Updatescited by 21
  34. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  35. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  36. 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
  37. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
  38. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19

50 most cited of 201 All 201 (JSON)