mlsanityData to 14 Oct 2026

USP: A Unified Sequence Parallelism Approach for Long Context Generative AI

arXiv 2405.07719 ↗First posted 13 May 2024cs.LG cs.AIcited by 61

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Sequence parallelism (SP), which divides the sequence dimension of input tensors across multiple computational devices, is becoming key to unlocking the long-context capabilities of generative AI models. This paper investigates the state-of-the-art SP approaches, i.e. DeepSpeed-Ulysses and Ring-Attention, and proposes a unified SP approach, which is more robust to transformer model architectures and network hardware topology. This paper compares the communication and memory cost of SP and existing parallelism, including data/tensor/zero/pipeline parallelism, and discusses the best practices for designing hybrid 4D parallelism involving SP. We achieved 47% MFU on two 8xA800 nodes using SP for the LLAMA3-8B model training using sequence length 208K. Our code is publicly available at https://github.com/feifeibear/long-context-attention.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Builds on 16 of its 22 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

16 of 22 references.

References · 16

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  9. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  10. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  11. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  12. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
4 more
  1. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  2. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  3. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
  4. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 21; 2026: 31.

61 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  3. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  4. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  6. 2025Test-Time Training Done Rightcited by 130
  7. 2025Scaling RL to Long Videoscited by 84
  8. 2024xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelismcited by 56
  9. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
  10. 2025Efficient Diffusion Models: A Surveycited by 35
  11. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 30
  12. 2024PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inferencecited by 28
38 more
  1. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  2. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  3. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 19
  4. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  5. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Trainingcited by 15
  6. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformerscited by 14
  7. 2026Action Images: End-to-End Policy Learning via Multiview Video Generationcited by 12
  8. 2025Domain Adaptation of Foundation LLMs for e-Commercecited by 9
  9. 2025Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulationcited by 8
  10. 2025SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperationcited by 7
  11. 2025DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelismcited by 7
  12. 2024MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Trainingcited by 5
  13. 2024PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inferencecited by 4
  14. 2025360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Trainingcited by 4
  15. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
  16. 2025Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequencescited by 3
  17. 2026Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoEcited by 3
  18. 2026M*: A Modular, Extensible, Serving System for Multimodal Modelscited by 3
  19. 2026LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budgetcited by 3
  20. 2025HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelismcited by 2
  21. 2025Accelerating Parallel Diffusion Model Serving with Residual Compressioncited by 2
  22. 2026SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUscited by 2
  23. 2026EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformerscited by 2
  24. 2026LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windowscited by 2
  25. 2026MoWorld: A Flash World Modelcited by 2
  26. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
  27. 2026Visko Orbis 1.0: A Live Model for Real-Time Interactive Long Video Generationcited by 2
  28. 2025Spatio-Temporal Parallelism for Diffusion Model Inference on Heterogeneous Multi-GPU Systemscited by 1
  29. 2025MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Trainingcited by 1
  30. 2025PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decouplingcited by 1
  31. 2026FastUSP: A Multi-Level Collaborative Acceleration Framework for Distributed Diffusion Model Inferencecited by 1
  32. 2026OSP-Next: Efficient High-Quality Video Generation with Sparse Sequence Parallelism, HiF8 Quantization, and Reinforcement Learningcited by 1
  33. 2026Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generationcited by 1
  34. 2026FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generationcited by 1
  35. 2026LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Timecited by 1
  36. 2026Miles v0.1: Production-Level Post-Trainingcited by 1
  37. 2026SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulatorscited by 1
  38. 2026Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generationcited by 1

50 most cited of 61 All 61 (JSON)