Data to 14 Oct 2026

Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization

arXiv 1506.08272 ↗First posted 27 Jun 2015math.OC cs.NA math.NAcited by 116

Not on the map. Why?

Abstract (arXiv)

Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, existing theories cannot explain their convergence and speedup properties, mainly due to the nonconvexity of most deep learning formulations and the asynchronous parallel mechanism. To fill the gaps in theory and provide theoretical supports, this paper studies two asynchronous parallel implementations of SG: one is on the computer network and the other is on the shared memory system. We establish an ergodic convergence rate O(1/√K) for both algorithms and prove that the linear speedup is achievable if the number of workers is bounded by √K (K is the total number of iterations). Our results generalize and improve existing analysis for convex minimization.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 32; 2022: 21; 2023: 16; 2024: 14; 2025: 15; 2026: 17.

116 papers.

Sort:

By area:

2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
2022Byzantine Machine Learning Made Easy by Resilient Averaging of Momentumscited by 63
2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
2021Decentralized Federated Learning: Balancing Communication and Computing Costscited by 55
2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
2022Multiplayer Performative Prediction: Learning in Decision-Dependent Gamescited by 44
2021Anarchic Federated Learningcited by 42
2021Secure Bilevel Asynchronous Vertical Federated Learning with Backward Updatingcited by 37
2021Mobility-Aware Cluster Federated Learning in Hierarchical Wireless Networkscited by 35
2021Asynchronous Stochastic Optimization Robust to Arbitrary Delayscited by 32

38 more

2021DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregationcited by 31
2021HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Frameworkcited by 28
2023DoCoFL: Downlink Compression for Cross-Device Federated Learningcited by 28
2022FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulationscited by 27
2024Asynchronous Local-SGD Training for Language Modelingcited by 26
2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
2022SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communicationcited by 21
2023Epidemic Learning: Boosting Decentralized Learning with Randomized Communicationcited by 21
2022AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregationcited by 19
2021Distributed Learning Systems with First-order Methodscited by 17
2021Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guaranteescited by 17
2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
2022A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updatescited by 17
2021Truly Sparse Neural Networks at Scalecited by 16
2023On the Role of Server Momentum in Federated Learningcited by 16
2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
2021Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGXcited by 14
2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
2021Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settingscited by 11
20211-Bit Compressive Sensing for Efficient Federated Learning Over the Aircited by 11
2023Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimizationcited by 11
2025Eager Updates For Overlapped Communication and Computation in DiLoCocited by 11
2022Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topologycited by 10
2021Distributed Deep Learning Using Volunteer Computing-Like Paradigmcited by 9
2022Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Datacited by 9
2021Learning Under Delayed Feedback: Implicitly Adapting to Gradient Delayscited by 7
2024Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clientscited by 7
2023Theoretically Principled Federated Learning for Balancing Privacy and Utilitycited by 6
2023Stochastic Unrolled Federated Learningcited by 6
2021The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communicationcited by 5
2022Semi-Synchronous Personalized Federated Learning over Mobile Edge Networkscited by 5
2023Linear Speedup of Incremental Aggregated Gradient Methods on Streaming Datacited by 5
2024Ordered Momentum for Asynchronous SGDcited by 5
2025ZenFlow: Enabling Stall-Free Offloading for LLM Trainingcited by 5
2025Ringleader ASGD: The First Asynchronous SGD with Optimal Time Complexity under Data Heterogeneitycited by 5
2021Parareal Neural Networks Emulating a Parallel-in-time Algorithmcited by 4
2021Collaborative Learning over Wireless Networks: An Introductory Overviewcited by 4
2020Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descentcited by 3

50 most cited of 116 All 116 (JSON)