mlsanityData to 7 Oct 2026

FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training

arXiv 2606.08476 ↗First posted 7 Jun 2026cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Context parallelism (CP) is essential for training large-scale, long-context language models, as it partitions sequences to reduce memory overhead. However, existing CP methods suffer from workload imbalance, inefficient kernels, and redundant communication due to static sequence sharding and key-value (KV) tensor communication. We present FlashCP, a load-balanced and communication-efficient framework for CP training. FlashCP introduces a sharding-aware communication mechanism to eliminate redundant KV communication and proposes a novel Whole-Doc sharding strategy that maximizes communication savings while maintaining balanced workloads. To efficiently combine Whole-Doc and Per-Doc sharding, FlashCP further designs a heuristic algorithm to search for near-optimal sharding plans. Extensive experiments show that FlashCP achieves up to 1.63x speedup over state-of-the-art CP frameworks across diverse datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 7 Jun 2026.

Builds on

22 of 34 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  11. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  12. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
10 more
  1. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  2. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  3. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  4. 2025WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Trainingcited by 17
  5. 2024LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelismcited by 16
  6. 2023DynaPipe: Optimizing Multi-task Training through Dynamic Pipelinescited by 12
  7. 2024Enhancing Training Efficiency Using Packing with Flash Attentioncited by 12
  8. 2024Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuningcited by 11
  9. 2025SkyLadder: Better and Faster Pretraining via Context Window Schedulingcited by 6
  10. 2025LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuningcited by 2

Built on by

Not cited yet.