mlsanityData to 14 Oct 2026

Robustifying Asynchronous SGD via Soft Throttling

arXiv 2609.39357 ↗First posted 30 Sep 2026cs.LG cs.DC math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Asynchronous SGD is a popular algorithm for distributed learning where each client's gradient update is applied on arrival. This leads to a speed-up, but also an increased vulnerability to attacks, as fast clients can dominate the total update. We introduce Throttle, a Byzantine-robust generalization of asynchronous SGD where the key idea is to exponentially down-weight updates from faster clients by a factor q. Both asynchronous SGD (q=1) and synchronous Byzantine-robust SGD (q→∞) correspond to specific settings of Throttle. We provide a theoretical analysis of the convergence rate and validate the robustness to attacks both theoretically and empirically. Remarkably, our experiments show that this down-weighting mechanism can also improve performance over standard asynchronous SGD even in the non-Byzantine setting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 61 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

41 of 61 references.

References · 41

  1. 2019Advances and Open Problems in Federated Learningcited by 3,046
  2. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  3. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  4. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  5. 2018The Hidden Vulnerability of Distributed Learning in Byzantiumcited by 329
  6. 2019Robust Aggregation for Federated Learningcited by 317
  7. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 288
  8. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  9. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  10. 2019A Survey on Distributed Machine Learningcited by 188
  11. 2017Revisiting Distributed Synchronous SGDcited by 142
  12. 2019Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulationcited by 141
29 more
  1. 2020Learning from History for Byzantine Robust Optimizationcited by 137
  2. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 128
  3. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 116
  4. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  5. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
  6. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  7. 2023Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneitycited by 65
  8. 2022Byzantine Machine Learning Made Easy by Resilient Averaging of Momentumscited by 63
  9. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
  10. 2019Zeno++: Robust Fully Asynchronous SGDcited by 53
  11. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  12. 2022Byzantine-Robust Federated Learning with Optimal Statistical Rates and Privacy Guaranteescited by 30
  13. 2022AFLGuard: Byzantine-robust Asynchronous Federated Learningcited by 29
  14. 2023Robust Distributed Learning: Tight Error Bounds and Breakdown Point under Data Heterogeneitycited by 26
  15. 2023Optimal Time Complexities of Parallel Stochastic Optimization Methods Under a Fixed Computation Modelcited by 24
  16. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  17. 2025Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexitycited by 18
  18. 2023AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithmscited by 16
  19. 2024Byzantine-Robust Federated Learning: Impact of Client Subsampling and Local Updatescited by 14
  20. 2024Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneitycited by 13
  21. 2023Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Ratescited by 12
  22. 2023On Convergence of Incremental Gradient for Non-Convex Smooth Functionscited by 7
  23. 2025Optimal Complexity in Byzantine-Robust Distributed Stochastic Optimization with Data Heterogeneitycited by 6
  24. 2023Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differencescited by 5
  25. 2024Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex Finite-Sum Optimization with Heterogeneous Asynchronous Computationscited by 5
  26. 2024Ordered Momentum for Asynchronous SGDcited by 5
  27. 2020Buffered Asynchronous SGD for Byzantine Learningcited by 4
  28. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  29. 2025Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous MLcited by 2

Built on by

Not cited yet.