mlsanityData to 7 Oct 2026

A Quadratic Synchronization Rule for Distributed Deep Learning

arXiv 2310.14423 ↗First posted 22 Oct 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In distributed deep learning with data parallelism, synchronizing gradients at each training step can cause a huge communication overhead, especially when many nodes work together to train large models. Local gradient methods, such as Local SGD, address this issue by allowing workers to compute locally for H steps without synchronizing with others, hence reducing communication frequency. While H has been viewed as a hyperparameter to trade optimization efficiency for communication cost, recent research indicates that setting a proper H value can lead to generalization improvement. Yet, selecting a proper H is elusive. This work proposes a theory-grounded method for determining H, named the Quadratic Synchronization Rule (QSR), which recommends dynamically setting H in proportion to frac1η2 as the learning rate η decays over time. Extensive ImageNet experiments on ResNet and ViT show that local gradient methods with QSR consistently improve the test accuracy over other synchronization strategies. Compared with the standard data parallel training, QSR enables Local AdamW on ViT-B to cut the training time on 16 or 64 GPUs down from 26.7 to 20.2 hours or from 8.6 to 5.5 hours and, at the same time, achieves 1.16% or 0.84% higher top-1 validation accuracy.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 87 of its 107 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

87 of 107 references.

References · 87

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2019Advances and Open Problems in Federated Learningcited by 3,021
  10. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
38 more
  1. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  10. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  13. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  14. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  15. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  16. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  17. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  18. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  19. 2017Three Factors Influencing Minima in SGDcited by 270
  20. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  21. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  22. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  23. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  24. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  25. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  26. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  27. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  28. 2020Directional convergence and alignment in deep learningcited by 172
  29. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  30. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  31. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  32. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  33. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  34. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  35. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  36. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  37. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  38. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 7
  2. 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
  3. 2025CurvaDion: Curvature-Adaptive Distributed Orthonormalizationcited by 1
  4. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
  5. 2026AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronizationcited by 0