mlsanityData to 14 Oct 2026

Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers

arXiv 2606.13287 ↗First posted 11 Jun 2026cs.LG cs.DC math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

In modern machine learning, parallelization of training is an important strategy for increasing scale. Asynchronous stochastic gradient descent (ASGD), which maximizes the utilization of available hardware by avoiding waiting for slow workers. However, with constant step sizes, the convergence of ASGD is nonetheless affected negatively by slow workers due to large delays in updates. At the same time, it has been empirically observed in asynchronous training of deep learning models that gradient clipping "stabilizes" training. In this work, we provide a theoretical justification for this behavior, as we show that clipping removes the dependence of the maximum delay in the oracle complexity. We employ a sub-Weibull model of gradient noise which generalizes sub-Gaussian and sub-exponential distributions to more heavy-tailed distributions, motivated by empirical observations in deep learning. We show convergence in expectation, and the first time in asynchronous optimization, convergence with high probability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

34 of 61 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  4. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  5. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  6. 2019Asynchronous Federated Optimizationcited by 267
  7. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  8. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  9. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  10. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  11. 2017Revisiting Distributed Synchronous SGDcited by 142
  12. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
22 more
  1. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  2. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  3. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
  4. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  5. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  6. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  7. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  8. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  9. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
  10. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 49
  11. 2015Staleness-aware Async-SGD for Distributed Deep Learningcited by 43
  12. 2018Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regressioncited by 42
  13. 2015An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimizationcited by 38
  14. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  15. 2021Asynchronous Stochastic Optimization Robust to Arbitrary Delayscited by 32
  16. 2020High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noisecited by 31
  17. 2023Optimal Time Complexities of Parallel Stochastic Optimization Methods Under a Fixed Computation Modelcited by 24
  18. 2019Non-Gaussianity of Stochastic Gradient Noisecited by 22
  19. 2025Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexitycited by 18
  20. 2021Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guaranteescited by 17
  21. 2022Delay-adaptive step-sizes for asynchronous learningcited by 11
  22. 2016On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithmscited by 7

Built on by

Not cited yet.