Re-evaluating the Memory-balanced Pipeline Parallelism: BPipe
Not on the map. Why?
Abstract (arXiv)
Pipeline parallelism is an essential technique in the training of large-scale Transformer models. However, it suffers from imbalanced memory consumption, leading to insufficient memory utilization. The BPipe technique was proposed to address this issue and has proven effective in the GPT-3 model. Nevertheless, our experiments have not yielded similar benefits for LLaMA training. Additionally, BPipe only yields negligible benefits for GPT-3 training when applying flash attention. We analyze the underlying causes of the divergent performance of BPipe on GPT-3 and LLaMA. Furthermore, we introduce a novel method to estimate the performance of BPipe.
Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 4 Jan 2024.
Builds on
9 of 15 references.
References · 9
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
- 2017Mixed Precision Trainingcited by 1,147
- 2016Training Deep Nets with Sublinear Memory Costcited by 840
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 301
- 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
- 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.