mlsanityData to 14 Oct 2026

Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency

arXiv 2606.07881 ↗First posted 5 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines preserve forward/backward weight consistency but suffer from bubbles; asynchronous pipelines remove bubbles but introduce weight-version mismatch, typically requiring weight stashing, prediction, or correction mechanisms. We introduce PACI (Pipeline Asynchronous training with Controlled Inconsistency), a bubble-free asynchronous pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization. The key idea is to use local gradient accumulation as a version-control mechanism: by slowing parameter-version evolution relative to pipeline delay, PACI limits the number of optimizer updates crossed by any micro-batch while preserving steady-state utilization. In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, achieves fully utilized pipeline throughput, and improves training time-to-accuracy by up to 1.69× over the fastest flush baseline. These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 29 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

24 of 29 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  8. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  10. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  11. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  12. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
12 more
  1. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  2. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  3. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  4. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  5. 2023Zero Bubble Pipeline Parallelismcited by 40
  6. 2018Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platformcited by 28
  7. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  8. 2019XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Trainingcited by 20
  9. 2024Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Trainingcited by 11
  10. 2025Nesterov Method for Asynchronous Pipeline Parallel Optimizationcited by 6
  11. 2023PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Predictioncited by 4
  12. 2026AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Trainingcited by 1

Built on by

Not cited yet.