mlsanityData to 7 Oct 2026

Decentralized SGD and Average-direction SAM are Asymptotically Equivalent

arXiv 2306.02913 ↗First posted 5 Jun 2023cs.LG cs.CY cs.DCcited by 15

Not on the map. Why?

Abstract (arXiv)

Decentralized stochastic gradient descent (D-SGD) allows collaborative learning on massive devices simultaneously without the control of a central server. However, existing theories claim that decentralization invariably undermines generalization. In this paper, we challenge the conventional belief and present a completely new perspective for understanding decentralized learning. We prove that D-SGD implicitly minimizes the loss function of an average-direction Sharpness-aware minimization (SAM) algorithm under general non-convex non-β-smooth settings. This surprising asymptotic equivalence reveals an intrinsic regularization-optimization trade-off and three advantages of decentralization: (1) there exists a free uncertainty evaluation mechanism in D-SGD to improve posterior estimation; (2) D-SGD exhibits a gradient smoothing effect; and (3) the sharpness regularization effect of D-SGD does not decrease as total batch size increases, which justifies the potential generalization benefit of D-SGD over centralized SGD (C-SGD) in large-batch scenarios. The code is available at https://github.com/Raiden-Zhu/ICML-2023-DSGD-and-SAM.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 89 of its 143 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

89 of 143 references.

References · 89

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2021Convex optimizationcited by 2,479
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  12. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
38 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2020Adaptive Federated Optimizationcited by 1,036
  3. 2019Deep Leakage from Gradientscited by 1,004
  4. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  5. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  9. 2017Exploring Generalization in Deep Learningcited by 595
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  13. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  14. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  15. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  16. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  17. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  18. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  19. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  20. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  21. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  22. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  23. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  24. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  25. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  26. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  27. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  28. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  29. 2018D2: Decentralized Training over Decentralized Datacited by 140
  30. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  31. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
  32. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  33. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  34. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  35. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  36. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  37. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  38. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 5; 2025: 3; 2026: 2.

15 papers.

Sort:

  1. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  2. 2023Layer-wise Linear Mode Connectivitycited by 22
  3. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  4. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  5. 2023Lookaround Optimizer: k steps around, 1 step averagecited by 9
  6. 2023DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learningcited by 6
  7. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  8. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  9. 2024On the Trade-off between Flatness and Optimization in Distributed Learningcited by 3
  10. 2025DICE: Data Influence Cascade in Decentralized Learningcited by 2
  11. 2024SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Datacited by 0
  12. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
3 more
  1. 2025On the Escaping Efficiency of Distributed Adversarial Training Algorithmscited by 0
  2. 2026Decentralized SGD with Controlled Disagreement Finds Flatter Minimacited by 0
  3. 2026Improved Convergence Analysis of Topology Dependence in Decentralized SGDcited by 0