# LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism

arXiv: https://arxiv.org/abs/2406.18485 · HTML: https://mlsanity.com/paper/2406.18485/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2024.

## Built on by (most cited first, 17 of 17)

- [LongVILA: Scaling Long-Context Visual Language Models for Long Videos](https://mlsanity.com/paper/2408.10188/) (2024)
- [MAGI-1: Autoregressive Video Generation at Scale](https://mlsanity.com/paper/2505.13211/) (2025)
- [Scaling RL to Long Videos](https://mlsanity.com/paper/2507.07966/) (2025)
- [xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism](https://mlsanity.com/paper/2411.01738/) (2024)
- [LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation](https://mlsanity.com/paper/2605.18739/) (2026)
- [DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training](https://mlsanity.com/paper/2502.07590/) (2025)
- [360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training](https://mlsanity.com/paper/2505.22296/) (2025)
- [DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training](https://mlsanity.com/paper/2601.21824/) (2026)
- [Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment](https://mlsanity.com/paper/2412.07894/) (2024)
- [Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences](https://mlsanity.com/paper/2506.13996/) (2025)
- [LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget](https://mlsanity.com/paper/2607.14952/) (2026)
- [MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training](https://mlsanity.com/paper/2510.18830/) (2025)
- [Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism](https://mlsanity.com/paper/2511.06247/) (2025)
- [Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking](https://mlsanity.com/paper/2602.21196/) (2026)
- [FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training](https://mlsanity.com/paper/2606.08476/) (2026)
- [Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training](https://mlsanity.com/paper/2609.19242/) (2026)
- [Memory-Efficient Expert Routing for Distributed MoE Training](https://mlsanity.com/paper/2610.07333/) (2026)

## Abstract (arXiv)

Efficiently training LLMs with long sequences is important yet challenged by the massive computation and memory requirements. Sequence parallelism has been proposed to tackle these problems, but existing methods suffer from scalability or efficiency issues. We propose LoongTrain, a novel system to efficiently train LLMs with long sequences at scale. The core of LoongTrain is the 2D-Attention mechanism, which combines both head-parallel and context-parallel techniques to break the scalability constraints while maintaining efficiency. We introduce Double-Ring-Attention and analyze the performance of device placement strategies to further speed up training. We implement LoongTrain with the hybrid ZeRO and Selective Checkpoint++ techniques. Experiment results show that LoongTrain outperforms state-of-the-art baselines, i.e., DeepSpeed-Ulysses and Megatron Context Parallelism, in both end-to-end training speed and scalability, and improves Model FLOPs Utilization (MFU) by up to 2.88x.