mlsanityData to 14 Oct 2026

BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training

arXiv 2610.14935 ↗First posted 10 Oct 2026cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Pipeline parallelism has become a critical technique for training large language models. To address the throughput limitation of existing sequence-level methods that schedule subsequences unidirectionally, we propose bidirectional sequence-level interleaved pipeline parallelism (BSIPipe), which co-schedules two opposing V-shaped interleaved pipelines at the subsequence level. To bound the activation accumulation from subsequence scaling, BSIPipe introduces a basic-unit concatenation mechanism that keeps activation memory invariant to the subsequence count. To eliminate communication deadlock risk from bidirectional scheduling, BSIPipe employs a communication optimization strategy that combines topology-aware device mapping with communication-computation overlap. In addition, BSIPipe composes with tensor parallelism to overcome the memory limitation of single-dimensional parallelism, neutralizing its bidirectional memory overhead while maintaining throughput. Experiments on GPT and LLaMA3 demonstrate that BSIPipe achieves average speedups of 1.11× to 1.13× in long-sequence training over SOTA sequence-level baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

16 of 28 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  8. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  9. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  10. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  11. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  12. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
4 more
  1. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  2. 2025SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Trainingcited by 6
  3. 2024BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Trainingcited by 3
  4. 2025TawPipe: Topology-Aware Weight Pipeline Parallelism for Accelerating Long-Context Large Models Trainingcited by 2

Built on by

Not cited yet.