mlsanityData to 7 Oct 2026

PipeDream: Fast and Efficient Pipeline Parallel DNN Training

arXiv 1806.03377 ↗First posted 8 Jun 2018cs.DCcited by 116

Not on the map. Why?

Abstract (arXiv)

PipeDream is a Deep Neural Network(DNN) training system for GPUs that parallelizes computation by pipelining execution across multiple machines. Its pipeline parallel computing model avoids the slowdowns faced by data-parallel training when large models and/or limited network bandwidth induce high communication-to-computation ratios. PipeDream reduces communication by up to 95% for large DNNs relative to data-parallel training, and allows perfect overlap of communication and computation. PipeDream keeps all available GPUs productive by systematically partitioning DNN layers among them to balance work and minimize communication, versions model parameters for backward pass correctness, and schedules the forward and backward passes of different inputs in round-robin fashion to optimize "time to target accuracy". Experiments with five different DNNs on two different clusters show that PipeDream is up to 5x faster in time-to-accuracy compared to data-parallel training.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 20; 2022: 11; 2023: 20; 2024: 24; 2025: 28; 2026: 12.

116 papers.

Sort:

By area:

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  5. 2024DeepSeek-V3 Technical Reportcited by 724
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  8. 2025Kimi K2: Open Agentic Intelligencecited by 523
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  10. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  12. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
38 more
  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  3. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  4. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  5. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  7. 2025Kling-Omni Technical Reportcited by 60
  8. 2025Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Modelscited by 58
  9. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 50
  10. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  11. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  12. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  13. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  14. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  15. 2023Zero Bubble Pipeline Parallelismcited by 39
  16. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
  17. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  18. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  19. 2021Maximizing Parallelism in Distributed Training for Huge Neural Networkscited by 23
  20. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  21. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  22. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  23. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  24. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 15
  25. 2021Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGXcited by 14
  26. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
  27. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13
  28. 2024Pipeline Parallelism with Controllable Memorycited by 11
  29. 2021AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learningcited by 8
  30. 2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
  31. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 8
  32. 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
  33. 2024Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studiescited by 7
  34. 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
  35. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 7
  36. 2021Automatic Graph Partitioning for Very Large-scale Deep Learningcited by 6
  37. 2021Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Trainingcited by 6
  38. 2024Activations and Gradients Compression for Model-Parallel Trainingcited by 6

50 most cited of 116 All 116 (JSON)