mlsanityData to 7 Oct 2026

BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training

arXiv 2609.03151 ↗First posted 2 Sep 2026cs.DC cs.LGcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Long-context reasoning for large language models (LLMs) is becoming increasingly important, but training over long sequences remains challenging due to massive memory and communication requirements. Sequence parallelism has emerged as an essential technique for addressing bottlenecks in long sequence LLM training. However, we observe that existing sequence parallelism methods are batch-agnostic and apply uniform sequence partitioning across all batch sizes, resulting in inefficient communication. In this paper, we introduce Batch- Aware Sequence Parallelism (BASP), a sequence parallelism approach that leverages batch structure to reduce communication overhead. BASP exploits batch structure by partitioning GPUs into disjoint sequence-parallel groups according to the micro- batch size. This design reduces the all-to-all communication group size, thereby localizing communication and improving training efficiency. Experimental results on an NVIDIA A100 cluster show that BASP improves end-to-end training time by up to 1.17 - 1.31x in Llama and Qwen models compared to standard sequence parallel baselines, while preserving identical model accuracy and memory usage.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 25 of its 30 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

25 of 30 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  12. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
13 more
  1. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  2. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  3. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  4. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 58
  5. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 52
  6. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  7. 2022MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloudcited by 21
  8. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  9. 2025ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUscited by 10
  10. 2024Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformercited by 6
  11. 2024PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Trainingcited by 5
  12. 2025HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelismcited by 2
  13. 2026TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Trainingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026NSP: Accelerating Variable-Length LLM Training via Nested Sequence Parallelism ↗
  2. 2026Efficient Scaling of LLM Training with Flexible Context Parallelism
  3. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  4. 2025StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMs
  5. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
  6. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  7. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  8. 2025AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design