mlsanityData to 14 Oct 2026

Breadth-First Pipeline Parallelism

arXiv 2211.05953 ↗First posted 11 Nov 2022cs.DC cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce Breadth-First Pipeline Parallelism, a novel training schedule which optimizes the combination of pipeline and data parallelism. Breadth-First Pipeline Parallelism lowers training time, cost and memory usage by combining a high GPU utilization with a small batch size per GPU, and by making use of fully sharded data parallelism. Experimentally, we observed an increase of up to 43% in training throughput for a 52 billion-parameter model using a small batch size per GPU compared to Megatron-LM, which would reduce the training time and cost by the same amount on a large GPU cluster.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 16 of its 21 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

16 of 21 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  10. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  11. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  12. 2018An Empirical Model of Large-Batch Trainingcited by 281
4 more
  1. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  2. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  3. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  4. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  3. 2024Scaling Deep Learning Training with MPMD Pipeline Parallelismcited by 1