mlsanityData to 14 Oct 2026

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

arXiv 2309.14509 ↗First posted 25 Sep 2023cs.LG cs.CL cs.DCcited by 245

Not on the map. Why?

Cited by 43 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Computation in a typical Transformer-based large language model (LLM) can be characterized by batch size, hidden dimension, number of layers, and sequence length. Until now, system works for accelerating LLM training have focused on the first three dimensions: data parallelism for batch size, tensor parallelism for hidden size and pipeline parallelism for model depth or layers. These widely studied forms of parallelism are not targeted or optimized for long sequence Transformer models. Given practical application needs for long sequence LLM, renewed attentions are being drawn to sequence parallelism. However, existing works in sequence parallelism are constrained by memory-communication inefficiency, limiting their scalability to long sequence large models. In this work, we introduce DeepSpeed-Ulysses, a novel, portable and effective methodology for enabling highly efficient and scalable LLM training with extremely long sequence length. DeepSpeed-Ulysses at its core partitions input data along the sequence dimension and employs an efficient all-to-all collective communication for attention computation. Theoretical communication analysis shows that whereas other methods incur communication overhead as sequence length increases, DeepSpeed-Ulysses maintains constant communication volume when sequence length and compute devices are increased proportionally. Furthermore, experimental evaluations show that DeepSpeed-Ulysses trains 2.5x faster with 4x longer sequence length than the existing method SOTA baseline.

Key facts. Cited by 245 papers in the map (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

21 of 31 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2020Rethinking Attention with Performerscited by 1,601
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  12. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
9 more
  1. 2023ClimaX: A foundation model for weather and climatecited by 337
  2. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  3. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  4. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  5. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  6. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  7. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  8. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 69
  9. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 50; 2025: 90; 2026: 98.

245 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  3. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  7. 2025Kimi-VL Technical Reportcited by 388
  8. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  9. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  10. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  11. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  12. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 259
38 more
  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  2. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 229
  3. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  4. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  5. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 168
  6. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  7. 2026Advancing Open-source World Modelscited by 154
  8. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  9. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  10. 2026Kimi K3: Open Frontier Intelligencecited by 138
  11. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  12. 2025Test-Time Training Done Rightcited by 130
  13. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  14. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  15. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  16. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  17. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  18. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 100
  19. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 90
  20. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 86
  21. 2025Scaling RL to Long Videoscited by 84
  22. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 83
  23. 2025Seaweed-7B: Cost-Effective Training of Video Generation Foundation Modelcited by 82
  24. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80
  25. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  26. 2025Kling-Omni Technical Reportcited by 71
  27. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  28. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 70
  29. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  30. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  31. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  32. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 61
  33. 2025Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Modelscited by 60
  34. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  35. 2024xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelismcited by 56
  36. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  37. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 47
  38. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45

50 most cited of 245 All 245 (JSON)