# Zero Bubble Pipeline Parallelism

arXiv: https://arxiv.org/abs/2401.10241 · HTML: https://mlsanity.com/paper/2401.10241/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022)
- [PipeDream: Fast and Efficient Pipeline Parallel DNN Training](https://mlsanity.com/paper/1806.03377/) (2018)
- [DAPPLE: A Pipelined Data Parallel Approach for Training Large Models](https://mlsanity.com/paper/2007.01045/) (2020)
- [MLIR: A Compiler Infrastructure for the End of Moore's Law](https://mlsanity.com/paper/2002.11054/) (2020)
- [PipeMare: Asynchronous Pipeline Parallel DNN Training](https://mlsanity.com/paper/1910.05124/) (2019)
- [Relay: A New IR for Machine Learning Frameworks](https://mlsanity.com/paper/1810.00952/) (2018)

## Built on by (most cited first, 30 of 40)

- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Defeating the Training-Inference Mismatch via FP16](https://mlsanity.com/paper/2510.26788/) (2025)
- [Optimizing RLHF Training for Large Language Models with Stage Fusion](https://mlsanity.com/paper/2409.13221/) (2024)
- [Scalable Training of Mixture-of-Experts Models with Megatron Core](https://mlsanity.com/paper/2603.07685/) (2026)
- [Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs](https://mlsanity.com/paper/2502.12982/) (2025)
- [Pipeline Parallelism with Controllable Memory](https://mlsanity.com/paper/2405.15362/) (2024)
- [Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning](https://mlsanity.com/paper/2408.14158/) (2024)
- [Nesterov Method for Asynchronous Pipeline Parallel Optimization](https://mlsanity.com/paper/2505.01099/) (2025)
- [PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization](https://mlsanity.com/paper/2503.01328/) (2025)
- [MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training](https://mlsanity.com/paper/2408.04307/) (2024)
- [Ferret: An Efficient Online Continual Learning Framework under Varying Memory Constraints](https://mlsanity.com/paper/2503.12053/) (2025)
- [Data movement limits to frontier model training](https://mlsanity.com/paper/2411.01137/) (2024)
- [Improving training time and GPU utilization in geo-distributed language model training](https://mlsanity.com/paper/2411.14458/) (2024)
- [A Review of DeepSeek Models' Key Innovative Techniques](https://mlsanity.com/paper/2503.11486/) (2025)
- [DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models](https://mlsanity.com/paper/2507.09955/) (2025)
- [AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism](https://mlsanity.com/paper/2601.22442/) (2026)
- [Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inference](https://mlsanity.com/paper/2407.19775/) (2024)
- [Echo: Simulating Distributed Training At Scale](https://mlsanity.com/paper/2412.12487/) (2024)
- [CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems](https://mlsanity.com/paper/2508.11287/) (2025)
- [BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models](https://mlsanity.com/paper/2512.12131/) (2025)
- [A Scalable Distributed Framework for Multimodal GigaVoxel Image Registration](https://mlsanity.com/paper/2509.25044/) (2025)
- [Fine-grained MoE Load Balancing with Linear Programming](https://mlsanity.com/paper/2511.16947/) (2025)
- [JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency](https://mlsanity.com/paper/2604.03044/) (2026)
- [BEAM: Bi-level Memory-adaptive Algorithmic Evolution for LLM-Powered Heuristic Design](https://mlsanity.com/paper/2604.12898/) (2026)
- [Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered](https://mlsanity.com/paper/2605.15622/) (2026)
- [2BP: 2-Stage Backpropagation](https://mlsanity.com/paper/2405.18047/) (2024)
- [FreeRide: Harvesting Bubbles in Pipeline Parallelism](https://mlsanity.com/paper/2409.06941/) (2024)

## Abstract (arXiv)

Pipeline parallelism is one of the key components for large-scale distributed training, yet its efficiency suffers from pipeline bubbles which were deemed inevitable. In this work, we introduce a scheduling strategy that, to our knowledge, is the first to successfully achieve zero pipeline bubbles under synchronous training semantics. The key idea behind this improvement is to split the backward computation into two parts, one that computes gradient for the input and another that computes for the parameters. Based on this idea, we handcraft novel pipeline schedules that significantly outperform the baseline methods. We further develop an algorithm that automatically finds an optimal schedule based on specific model configuration and memory limit. Additionally, to truly achieve zero bubble, we introduce a novel technique to bypass synchronizations during the optimizer step. Experimental evaluations show that our method outperforms the 1F1B schedule up to 23% in throughput under a similar memory limit. This number can be further pushed to 31% when the memory constraint is relaxed. We believe our results mark a major step forward in harnessing the true potential of pipeline parallelism. We open sourced our implementation based on the popular Megatron-LM repository on https://github.com/sail-sg/zero-bubble-pipeline-parallelism.