mlsanityData to 7 Oct 2026

Parallel SGD: When does averaging help?

arXiv 1606.07365 ↗First posted 23 Jun 2016stat.ML cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Consider a number of workers running SGD independently on the same pool of data and averaging the models every once in a while -- a common but not well understood practice. We study model averaging as a variance-reducing mechanism and describe two ways in which the frequency of averaging affects convergence. For convex objectives, we show the benefit of frequent averaging depends on the gradient variance envelope. For non-convex objectives, we illustrate that this benefit depends on the presence of multiple globally optimal points. We complement our findings with multicore experiments on both synthetic and real data.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 3; 2023: 5; 2024: 10; 2025: 1.

30 papers.

Sort:

  1. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  2. 2021FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimizationcited by 29
  3. 2021Accelerating Gossip SGD with Periodic Global Averagingcited by 25
  4. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  5. 2022On the Performance of Gradient Tracking with Local Updatescited by 21
  6. 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
  7. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
  8. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  9. 2024CO2: Efficient Distributed Training with Full Communication-Computation Overlapcited by 17
  10. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  11. 2021Truly Sparse Neural Networks at Scalecited by 15
  12. 2023Accelerating Hybrid Federated Learning Convergence under Partial Participationcited by 15
18 more
  1. 2024The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communicationcited by 12
  2. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  3. 2021Local SGD Optimizes Overparameterized Neural Networks in Polynomial Timecited by 10
  4. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
  5. 2024DrJAX: Scalable and Differentiable MapReduce Primitives in JAXcited by 9
  6. 2023Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Datacited by 7
  7. 2021The Minimax Complexity of Distributed Optimizationcited by 6
  8. 2024PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory Systemcited by 5
  9. 2024FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantizationcited by 5
  10. 2023Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beatcited by 4
  11. 2023Hierarchical Weight Averaging for Deep Neural Networkscited by 2
  12. 2022Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGDcited by 1
  13. 2024Communication-Efficient Distributed Deep Learning via Federated Dynamic Averagingcited by 1
  14. 2021CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Mannercited by 0
  15. 2023FedDec: Peer-to-peer Aided Federated Learningcited by 0
  16. 2024A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniquescited by 0
  17. 2024EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Modelscited by 0
  18. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0