mlsanityData to 14 Oct 2026

Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training

arXiv 1811.03619 ↗First posted 8 Nov 2018cs.LG cs.DC stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which are communicated to the parameter server while updated parameters are returned. Recently, distributed training with AllReduce operations gained popularity as well. While many of those operations seem appealing, little is reported about wall-clock training time improvements. In this paper, we carefully analyze the AllReduce based setup, propose timing models which include network latency, bandwidth, cluster size and compute time, and demonstrate that a pipelined training with a width of two combines the best of both synchronous and asynchronous training. Specifically, for a setup consisting of a four-node GPU cluster we show wall-clock time training improvements of up to 5.4x compared to conventional approaches.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 8; 2023: 3; 2024: 6; 2025: 2; 2026: 1.

28 papers.

Sort:

By area:

  1. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  2. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  3. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  4. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  5. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  6. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  7. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  8. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  9. 2021Distributed Learning Systems with First-order Methodscited by 17
  10. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  11. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  12. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
16 more
  1. 2023DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updatescited by 12
  2. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  3. 2024MSPipe: Efficient Temporal GNN Training via Staleness-Aware Pipelinecited by 12
  4. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
  5. 2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
  6. 2022Personalized Federated Learning with Communication Compressioncited by 6
  7. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
  8. 2023FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Trainingcited by 4
  9. 2021LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Schedulingcited by 2
  10. 2021EventGraD: Event-Triggered Communication in Parallel Machine Learningcited by 1
  11. 2022Locally Asynchronous Stochastic Gradient Descent for Decentralised Deep Learningcited by 1
  12. 2022Reducing Training Time in Cross-Silo Federated Learning using Multigraph Topologycited by 1
  13. 2021CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensationcited by 0
  14. 2024FedEx: Expediting Federated Learning over Heterogeneous Mobile Devices by Overlapping and Participant Selectioncited by 0
  15. 2024On the Convergence Theory of Pipeline Gradient-based Analog In-memory Trainingcited by 0
  16. 2026One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretrainingcited by 0