mlsanityData to 14 Oct 2026

A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability

arXiv 2605.18750 ↗First posted 18 May 2026cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Pipeline parallelism is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in computation and communication. Existing pipeline systems typically consume static, profiled, or adaptively generated schedules as pre-committed execution orders. When realized task readiness diverges from the pre-committed order, stages may wait for not-yet-ready work even though other executable work is available, creating stage misalignment, idle bubbles, and reduced utilization. We present Runtime-Readiness-First Pipeline (RRFP), a readiness-driven runtime for pipeline-parallel training. RRFP changes how schedules are consumed at runtime: instead of treating a schedule as a sequence that stages must wait to follow, it treats the schedule as a non-binding hint order for ranking currently ready work. To support this model, RRFP combines message-driven asynchronous communication, lightweight tensor-parallel coordination for collective consistency, and ready-set arbitration for low-overhead dispatch. We implement RRFP in a Megatron-based training framework and evaluate it on language-only and multimodal workloads at up to 128 GPUs. RRFP improves over fixed-order pipeline baselines across all settings. Using the BFW hint, RRFP achieves up to 1.77× speedup on language-only workloads and up to 2.77× on multimodal workloads. In cross-framework comparisons, RRFP with the default BF hint outperforms the faster available external system by up to 1.84× while preserving training correctness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

31 of 48 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  11. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  12. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
19 more
  1. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  2. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  3. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  4. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  5. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  6. 2025Parallel Scaling Law for Language Modelscited by 30
  7. 2024Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitationcited by 14
  8. 2019Dynamic Stale Synchronous Parallel Distributed Training for Deep Learningcited by 12
  9. 2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
  10. 2024Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Trainingcited by 11
  11. 2024GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelismcited by 9
  12. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusterscited by 9
  13. 2024Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Schedulingcited by 7
  14. 2025SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchipscited by 6
  15. 2025Efficient Distributed MLLM Training with Cornstarchcited by 4
  16. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  17. 2025WATOS: Efficient LLM Training Strategies and Architecture Co-exploration for Wafer-scale Chipcited by 2
  18. 2020PipeTune: Pipeline Parallelism of Hyper and System Parameters Tuning for Deep Learning Clusterscited by 1
  19. 2025A Flexible Programmable Pipeline Parallelism Framework for Efficient DNN Trainingcited by 1

Built on by

Not cited yet.