mlsanityData to 7 Oct 2026

Memory-Efficient Pipeline-Parallel DNN Training

arXiv 2006.09503 ↗First posted 16 Jun 2020cs.LG cs.DC stat.MLcited by 133

Not on the map. Why?

Abstract (arXiv)

Many state-of-the-art ML results have been obtained by scaling up the number of parameters in existing models. However, parameters and activations for such large models often do not fit in the memory of a single accelerator device; this means that it is necessary to distribute training of large models over multiple accelerators. In this work, we propose PipeDream-2BW, a system that supports memory-efficient pipeline parallelism. PipeDream-2BW uses a novel pipelining and weight gradient coalescing strategy, combined with the double buffering of weights, to ensure high throughput, low memory footprint, and weight update semantics similar to data parallelism. In addition, PipeDream-2BW automatically partitions the model over the available hardware resources, while respecting hardware constraints such as memory capacities of accelerators and interconnect topologies. PipeDream-2BW can accelerate the training of large GPT and BERT language models by up to 20× with similar final model accuracy.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 21; 2023: 26; 2024: 26; 2025: 26; 2026: 19.

133 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2023The Falcon Series of Open Language Modelscited by 526
  5. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  6. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  7. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  8. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  9. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  10. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  11. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  12. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
38 more
  1. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  2. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  3. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  4. 2025Kling-Omni Technical Reportcited by 60
  5. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  6. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 50
  7. 2024xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Proteincited by 49
  8. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  9. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  10. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  11. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  12. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  13. 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
  14. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  15. 2023Reducing Energy Bloat in Large Model Trainingcited by 22
  16. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  17. 2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
  18. 2022On Optimizing the Communication of Model Parallelismcited by 19
  19. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  20. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16
  21. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  22. 2024Forecasting GPU Performance for Deep Learning Training and Inferencecited by 15
  23. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 14
  24. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  25. 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13
  26. 2022Deep Incubation: Training Large Models by Divide-and-Conqueringcited by 13
  27. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  28. 2021Automap: Towards Ergonomic Automated Parallelism for ML Modelscited by 12
  29. 2021Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Trainingcited by 11
  30. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  31. 2023Does compressing activations help model parallel training?cited by 11
  32. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  33. 2024PartIR: Composing SPMD Partitioning Strategies for Machine Learningcited by 10
  34. 2024DeMo: Decoupled Momentum Optimizationcited by 10
  35. 2025Eager Updates For Overlapped Communication and Computation in DiLoCocited by 10
  36. 2024DiPaCo: Distributed Path Compositioncited by 9
  37. 2024GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelismcited by 9
  38. 2024Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inferencecited by 9

50 most cited of 133 All 133 (JSON)