mlsanityData to 14 Oct 2026

Sequence Parallelism: Long Sequence Training from System Perspective

arXiv 2105.13120 ↗First posted 26 May 2021cs.LG cs.DCcited by 126

Not on the map. Why?

Abstract (arXiv)

Transformer achieves promising results on various tasks. However, self-attention suffers from quadratic memory requirements with respect to the sequence length. Existing work focuses on reducing time and space complexity from an algorithm perspective. In this work, we propose sequence parallelism, a memory-efficient parallelism method to help us break input sequence length limitation and train with longer sequences on GPUs efficiently. Our approach is compatible with most existing parallelisms (e.g. data parallelism, pipeline parallelism and tensor parallelism), which means our sequence parallelism makes 4D parallelism possible. More importantly, we no longer require a single device to hold the whole sequence. That is, with sparse attention, our sequence parallelism enables us to train transformer with infinite long sequence. Specifically, we split the input sequence into multiple chunks and feed each chunk into its corresponding device (i.e. GPU). To compute the attention output, we integrated ring-style communication with self-attention calculation and proposed Ring Self-Attention (RSA). Experiments show that sequence parallelism performs well when scaling with batch size and sequence length. Compared with tensor parallelism, our approach achieved 13.7× and 3.0× maximum batch size and sequence length respectively when scaling up to 64 NVIDIA P100 GPUs. With sparse attention, sequence can handle sequence with over 114K tokens, which is over 27× longer than existing sparse attention works holding the whole sequence on a single device.

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 26 May 2021.

Builds on

21 of 38 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  11. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  12. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 235
9 more
  1. 2020ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 162
  2. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  3. 2020Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Poolingcited by 115
  4. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  5. 2019BERT with History Answer Embedding for Conversational Question Answeringcited by 67
  6. 2021Natural Language Video Localization: A Revisit in Span-based Question Answering Frameworkcited by 55
  7. 2020GDPNet: Refining Latent Multi-View Graph for Relation Extractioncited by 29
  8. 2020An Embarrassingly Simple Model for Dialogue Relation Extractioncited by 13
  9. 2019High Resolution Medical Image Analysis with Spatial Partitioningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 13; 2024: 38; 2025: 46; 2026: 27.

126 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2024Yi: Open Foundation Models by 01.AIcited by 719
  3. 2023The Falcon Series of Open Language Modelscited by 535
  4. 2024Long Context Transfer from Language to Visioncited by 515
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  7. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  8. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  9. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  10. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  12. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
38 more
  1. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 170
  2. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  3. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  4. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  5. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  6. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 96
  7. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  8. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  9. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  10. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 50
  11. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 48
  12. 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 40
  13. 2025HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generationcited by 39
  14. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  15. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  16. 2025In-Context Learning with Unpaired Clips for Instruction-based Video Editingcited by 35
  17. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
  18. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
  19. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 30
  20. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  21. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  22. 2025Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMscited by 26
  23. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  24. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
  25. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  26. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 16
  27. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  28. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  29. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Userscited by 15
  30. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  31. 2025Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devicescited by 15
  32. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  33. 2025Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUscited by 13
  34. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  35. 2024Scaling Granite Code Models to 128K Contextcited by 11
  36. 2024BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequencescited by 10
  37. 2024Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inferencecited by 9
  38. 2024Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Paralleliscited by 9

50 most cited of 126 All 126 (JSON)