mlsanityData to 7 Oct 2026

Decentralized SGD with Controlled Disagreement Finds Flatter Minima

arXiv 2602.02899 ↗First posted 2 Feb 2026cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization. This work challenges this view by introducing decentralized SGD with Adaptive Consensus (DSGD-AC), which uses a time-dependent scaling mechanism to maintain consensus errors throughout the training. We show that adaptive consensus changes the stationary variance of disagreement modes by balancing two effects: it preserves consensus-error magnitude through weaker graph damping while still allowing curvature-dependent damping to shape the disagreement directions. This balance can produce a stronger Hessian-weighted loss-envelope penalty around the deployed model, even when normalized Hessian alignment is weaker than in standard DSGD. Empirical results on image classification show that DSGD-AC reaches flatter solutions and higher test accuracy than standard DSGD and even centralized SGD. Together, these results support consensus errors as a useful implicit regularizer and open a new perspective on the design of decentralized learning algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2016Wide Residual Networkscited by 3,057
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  7. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  8. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  9. 2024The Road Less Scheduledcited by 158
  10. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  11. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  12. 2018Distributed Stochastic Gradient Tracking Methodscited by 106
20 more
  1. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  2. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  3. 2021Exponential Graph is Provably Efficient for Decentralized Deep Trainingcited by 55
  4. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  5. 2024Friendly Sharpness-Aware Minimizationcited by 52
  6. 2021Consensus Control for Decentralized Deep Learningcited by 49
  7. 2021A Unified and Refined Convergence Analysis for Non-Convex Decentralized Learningcited by 43
  8. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  9. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  10. 2022Topology-aware Generalization of Decentralized SGDcited by 37
  11. 2024Does SGD really happen in tiny subspaces?cited by 35
  12. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  13. 2022Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Datacited by 20
  14. 2023Decentralized Learning Made Easy with DecentralizePycited by 18
  15. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  16. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  17. 2023DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updatescited by 12
  18. 2023Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithmcited by 10
  19. 2023Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergencecited by 8
  20. 2024From promise to practice: realizing high-performance decentralized trainingcited by 4

Built on by

Not cited yet.