Data to 7 Oct 2026

Revisiting Distributed Synchronous SGD

arXiv 1702.05800 ↗First posted 19 Feb 2017cs.DC cs.AI cs.LGcited by 142

Not on the map. Why?

Abstract (arXiv)

Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 30; 2023: 23; 2024: 18; 2025: 24; 2026: 9.

142 papers.

Sort:

By area:

2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
2021PPFL: Privacy-preserving Federated Learning with Trusted Execution Environmentscited by 93
2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34

38 more

2022Distributed Graph Neural Network Training: A Surveycited by 32
2021FEDZIP: A Compression Framework for Communication-Efficient Federated Learningcited by 27
2021Resource-Efficient Federated Learningcited by 26
2023Lero: A Learning-to-Rank Query Optimizercited by 26
2024Decentralized Federated Learning: A Survey on Security and Privacycited by 25
2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
2022Scheduling and Aggregation Design for Asynchronous Federated Learning over Wireless Networkscited by 20
2024Communication Efficient Distributed Training with Distributed Lioncited by 18
2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
2024CO2: Efficient Distributed Training with Full Communication-Computation Overlapcited by 17
2025Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexitycited by 17
2023A Survey for Federated Learning Evaluations: Goals and Measurescited by 16
2021Differential Privacy Meets Federated Learning under Communication Constraintscited by 15
2021AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learningcited by 15
2021OneFlow: Redesign the Distributed Deep Learning Framework from Scratchcited by 15
2022Deep Reinforcement Learning Assisted Federated Learning Algorithm for Data Management of IIoTcited by 15
2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
2023The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Surveycited by 13
2022Federated Learning and Meta Learning: Approaches, Applications, and Directionscited by 12
2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
2023FedDefender: Client-Side Attack-Tolerant Federated Learningcited by 12
2025Understanding Stragglers in Large Model Training Using What-if Analysiscited by 12
2021Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settingscited by 11
2023Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimizationcited by 11
2021Distilling Large Language Models into Tiny and Effective Students using pQRNNcited by 10
2021Communication-Efficient Hierarchical Federated Learning for IoT Heterogeneous Systems with Imbalanced Datacited by 10
2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptationcited by 9
2021Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and modelcited by 8
2021Gradient Coding with Dynamic Clustering for Straggler-Tolerant Distributed Learningcited by 8
2021AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learningcited by 8
2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
2021Device Scheduling and Update Aggregation Policies for Asynchronous Federated Learningcited by 7
2021A Survey and Empirical Evaluation of Parallel Deep Learning Frameworkscited by 7
2023A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learningcited by 7
2022Accelerating Asynchronous Federated Learning Convergence via Opportunistic Mobile Relayingcited by 6
2022Boost Decentralized Federated Learning in Vehicular Networks by Diversifying Data Sourcescited by 5

50 most cited of 142 All 142 (JSON)