mlsanityData to 7 Oct 2026

Ring Attention with Blockwise Transformers for Near-Infinite Context

arXiv 2310.01889 ↗First posted 3 Oct 2023cs.CLcited by 448

Not on the map. Why?

Cited by 55 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability to handle long sequences, thereby posing challenges in utilizing videos, actions, and other long-form sequences and modalities in complex environments. We present a novel approach, Ring Attention with Blockwise Transformers (Ring Attention), which leverages blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices while fully overlapping the communication of key-value blocks with the computation of blockwise attention. Our approach enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers, without resorting to approximations or incurring additional communication and computation overheads. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of our approach in allowing millions of tokens context size and improving performance.

Key facts. Cited by 448 papers in the map (data through 7 Oct 2026). Builds on 21 of its 44 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

21 of 44 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  9. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  10. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  11. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  12. 2018Online normalizer calculation for softmaxcited by 154
9 more
  1. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  2. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  3. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  4. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  5. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  6. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  7. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  8. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  9. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 125; 2025: 183; 2026: 133.

448 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  9. 2024Long Context Transfer from Language to Visioncited by 502
  10. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  11. 2025Kimi-VL Technical Reportcited by 381
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
38 more
  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  2. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  3. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  5. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  6. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  7. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  8. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  9. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  11. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  12. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  13. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  14. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 157
  15. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  16. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  17. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  18. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  19. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  20. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  21. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  22. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  23. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  24. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 102
  25. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  26. 2026Kimi K3: Open Frontier Intelligencecited by 99
  27. 2024Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scalecited by 87
  28. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  29. 2025Scaling RL to Long Videoscited by 83
  30. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  31. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 80
  32. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  33. 2025LongCat-Video Technical Reportcited by 73
  34. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  35. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  36. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Frameworkcited by 70
  37. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  38. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68

50 most cited of 448 All 448 (JSON)