mlsanityData to 7 Oct 2026

Why (and When) does Local SGD Generalize Better than SGD?

arXiv 2303.01215 ↗First posted 2 Mar 2023cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Local SGD is a communication-efficient variant of SGD for large-scale training, where multiple GPUs perform SGD independently and average the model parameters periodically. It has been recently observed that Local SGD can not only achieve the design goal of reducing the communication overhead but also lead to higher test accuracy than the corresponding SGD baseline (Lin et al., 2020b), though the training regimes for this to happen are still in debate (Ortiz et al., 2021). This paper aims to understand why (and when) Local SGD generalizes better based on Stochastic Differential Equation (SDE) approximation. The main contributions of this paper include (i) the derivation of an SDE that captures the long-term behavior of Local SGD in the small learning rate regime, showing how noise drives the iterate to drift and diffuse after it has reached close to the manifold of local minima, (ii) a comparison between the SDEs of Local SGD and SGD, showing that Local SGD induces a stronger drift term that can result in a stronger effect of regularization, e.g., a faster reduction of sharpness, and (iii) empirical evidence validating that having a small learning rate and long enough training time enables the generalization improvement over SGD but removing either of the two conditions leads to no improvement.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 13 of its 82 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

13 of 82 references.

References · 13

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2017Three Factors Influencing Minima in SGDcited by 270
  4. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  5. 2017Searching for Activation Functionscited by 158
  6. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 36
  7. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
  8. 2015Experiments on Parallel Training of Deep Neural Network using Model Averagingcited by 14
  9. 2014Parallel training of DNNs with Natural Gradient and Parameter Averagingcited by 11
  10. 2018The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effectscited by 9
  11. 2017On the diffusion approximation of nonconvex stochastic gradient descentcited by 5
  12. 2021Exponential escape efficiency of SGD from sharp minima in non-stationary regimecited by 5
1 more
  1. 2016Revisiting Distributed Synchronous SGDcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 10; 2025: 5; 2026: 2.

26 papers.

Sort:

  1. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  2. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  3. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  4. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  5. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  6. 2023How to escape sharp minima with random perturbationscited by 13
  7. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  8. 2023Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!cited by 8
  9. 2023The Expressibility of Polynomial based Attention Schemecited by 7
  10. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 7
  11. 2026FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learningcited by 7
  12. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
14 more
  1. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  2. 2023Leveraging Function Space Aggregation for Federated Learning at Scalecited by 4
  3. 2024Differentiated Aggregation to Improve Generalization in Federated Learningcited by 4
  4. 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
  5. 2024Riemannian Federated Learning via Averaging Gradient Streamscited by 4
  6. 2024How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learningcited by 2
  7. 2025Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysiscited by 2
  8. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  9. 2024Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?cited by 0
  10. 2024EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Modelscited by 0
  11. 2025Convergence Analysis of Federated Learning Methods Using Backward Error Analysiscited by 0
  12. 2025Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Trainingcited by 0
  13. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
  14. 2026Second-Order Path Kernel Interpolation Formulas in Machine Learningcited by 0