mlsanityData to 7 Oct 2026

Asynchronous Stochastic Gradient Descent with Delay Compensation

arXiv 1609.08326 ↗First posted 27 Sep 2016cs.LG cs.DCcited by 98

Not on the map. Why?

Abstract (arXiv)

With the fast development of deep learning, it has become common to learn big neural networks using massive training data. Asynchronous Stochastic Gradient Descent (ASGD) is widely adopted to fulfill this task for its efficiency, which is, however, known to suffer from the problem of delayed gradients. That is, when a local worker adds its gradient to the global model, the global model may have been updated by other workers and this gradient becomes "delayed". We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD. This is achieved by leveraging Taylor expansion of the gradient function and efficient approximation to the Hessian matrix of the loss function. We call the new algorithm Delay Compensated ASGD (DC-ASGD). We evaluated the proposed algorithm on CIFAR-10 and ImageNet datasets, and the experimental results demonstrate that DC-ASGD outperforms both synchronous SGD and asynchronous SGD, and nearly approaches the performance of sequential SGD.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 25; 2022: 16; 2023: 15; 2024: 14; 2025: 11; 2026: 16.

98 papers.

Sort:

By area:

  1. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  2. 2022The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retrainingcited by 109
  3. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  4. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  5. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  6. 2022Federated Learning via Inexact ADMMcited by 44
  7. 2022AFLGuard: Byzantine-robust Asynchronous Federated Learningcited by 29
  8. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  9. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  10. 2022AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregationcited by 19
  11. 2021FedAR: Activity and Resource-Aware Federated Learning Model for Distributed Mobile Robotscited by 18
  12. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
38 more
  1. 2021Distributed Learning Systems with First-order Methodscited by 17
  2. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  3. 2021Towards Demystifying Serverless Machine Learning Trainingcited by 16
  4. 2023AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithmscited by 16
  5. 2023On the Role of Server Momentum in Federated Learningcited by 16
  6. 2021Truly Sparse Neural Networks at Scalecited by 15
  7. 2021Adaptive Transmission Scheduling in Wireless Networks for Asynchronous Federated Learningcited by 15
  8. 2021Communication-Efficient ADMM-based Federated Learningcited by 14
  9. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  10. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 13
  11. 2021Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimizationcited by 12
  12. 2023Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimizationcited by 11
  13. 2021FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximationcited by 10
  14. 2024Noiseless Privacy-Preserving Decentralized Learningcited by 9
  15. 2021Distributed Deep Learning Using Volunteer Computing-Like Paradigmcited by 8
  16. 2022FedGiA: An Efficient Hybrid Algorithm for Federated Learningcited by 8
  17. 2021Learning Under Delayed Feedback: Implicitly Adapting to Gradient Delayscited by 6
  18. 2022Accelerating Asynchronous Federated Learning Convergence via Opportunistic Mobile Relayingcited by 6
  19. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
  20. 2025Nesterov Method for Asynchronous Pipeline Parallel Optimizationcited by 6
  21. 2021AutoSpace: Neural Architecture Search with Less Human Interferencecited by 5
  22. 2024Ordered Momentum for Asynchronous SGDcited by 5
  23. 2025Ringleader ASGD: The First Asynchronous SGD with Optimal Time Complexity under Data Heterogeneitycited by 5
  24. 2021AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruningcited by 4
  25. 2021Collaborative Learning over Wireless Networks: An Introductory Overviewcited by 4
  26. 2022A review of Federated Learning in Intrusion Detection Systems for IoTcited by 4
  27. 2022GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Modelcited by 4
  28. 2023AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrixcited by 4
  29. 2024Resource Management for Low-latency Cooperative Fine-tuning of Foundation Models at the Network Edgecited by 4
  30. 2024Deep Fr'echet Regressioncited by 4
  31. 2025Ferret: An Efficient Online Continual Learning Framework under Varying Memory Constraintscited by 4
  32. 2020Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descentcited by 3
  33. 2021Guided parallelized stochastic gradient descent for delay compensationcited by 3
  34. 2022Energy Minimization for Federated Asynchronous Learning on Battery-Powered Mobile Devices via Application Co-runningcited by 3
  35. 2022Distributed Graph Neural Network Training with Periodic Stale Representation Synchronizationcited by 3
  36. 2023DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipeliningcited by 3
  37. 2023Asynchronous Multi-Model Dynamic Federated Learning over Wireless Networks: Theory, Modeling, and Optimizationcited by 3
  38. 2024Momentum Approximation in Asynchronous Private Federated Learningcited by 3

50 most cited of 98 All 98 (JSON)