mlsanityData to 7 Oct 2026

Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimization

arXiv 2311.00465 ↗First posted 1 Nov 2023math.OC cs.DC cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Decentralized and asynchronous communications are two popular techniques to speedup communication complexity of distributed machine learning, by respectively removing the dependency over a central orchestrator and the need for synchronization. Yet, combining these two techniques together still remains a challenge. In this paper, we take a step in this direction and introduce Asynchronous SGD on Graphs (AGRAF SGD) -- a general algorithmic framework that covers asynchronous versions of many popular algorithms including SGD, Decentralized SGD, Local SGD, FedBuff, thanks to its relaxed communication and computation assumptions. We provide rates of convergence under much milder assumptions than previous decentralized asynchronous works, while still recovering or even improving over the best know results for all the algorithms covered.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

27 of 48 references.

References · 27

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  3. 2014Convex Optimization: Algorithms and Complexitycited by 569
  4. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  5. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  6. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  7. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  8. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  9. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  10. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  11. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  12. 2017Revisiting Distributed Synchronous SGDcited by 142
15 more
  1. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  2. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
  3. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  4. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  5. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  6. 2020Advances in Asynchronous Parallel and Distributed Optimizationcited by 40
  7. 2023Stochastic Gradient Descent under Markovian Sampling Schemescited by 37
  8. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  9. 2019Asynchronous Decentralized SGD with Quantized and Local Updatescited by 35
  10. 2018Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimizationcited by 34
  11. 2022SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communicationcited by 21
  12. 2018Asynchronous Gradient-Pushcited by 20
  13. 2021Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guaranteescited by 17
  14. 2022Asynchronous Decentralized Federated Learning for Collaborative Fault Diagnosis of PV Stationscited by 9
  15. 2023Delay-agnostic Asynchronous Coordinate Update Algorithmcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 2.

11 papers.

  1. 2024Noiseless Privacy-Preserving Decentralized Learningcited by 9
  2. 2024Boosting Asynchronous Decentralized Learning with Model Fragmentationcited by 6
  3. 2024DRACO: Decentralized Asynchronous Federated Learning over Row-Stochastic Wireless Networkscited by 5
  4. 2024Asynchronous Federated Stochastic Optimization for Heterogeneous Objectives Under Arbitrary Delayscited by 2
  5. 2025DICE: Data Influence Cascade in Decentralized Learningcited by 2
  6. 2024Dual-Delayed Asynchronous SGD for Arbitrarily Heterogeneous Datacited by 1
  7. 2026Tight Analysis of Decentralized SGD: A Markov Chain Perspectivecited by 1
  8. 2024LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdatescited by 0
  9. 2025A Tale of Two Learning Algorithms: Multiple Stream Random Walk and Asynchronous Gossipcited by 0
  10. 2025Analysis of Asynchronous Federated Learning: Unraveling the Interactions between Gradient Compression, Delay, and Data Heterogeneitycited by 0
  11. 2026Improved Convergence Analysis of Topology Dependence in Decentralized SGDcited by 0