mlsanityData to 7 Oct 2026

World Model on Million-Length Video And Language With Blockwise RingAttention

arXiv 2402.08268 ↗First posted 13 Feb 2024cs.LGcited by 196

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

Enabling long-context understanding remains a key challenge in scaling existing sequence models -- a crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, we aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities in long video understanding. We detail our long context data curation process, progressive context extension from 4K to 1M tokens, and present an efficient open-source implementation for scalable training on long sequences. Additionally, we open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens.

Key facts. Cited by 196 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 48 of its 61 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

48 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2020Longformer: The Long-Document Transformercited by 3,362
35 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2022Video Diffusion Modelscited by 2,280
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  14. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  15. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  16. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  17. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  18. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  21. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  22. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  23. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  24. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  25. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  26. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  27. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  28. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  29. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  30. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  31. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 52
  32. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32
  33. 2024aMUSEd: An Open MUSE Reproductioncited by 32
  34. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12
  35. 2023GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Lengthcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 91; 2026: 57.

196 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  4. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  5. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  6. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  8. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  9. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  10. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  11. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  12. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
38 more
  1. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  2. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  3. 2024Make Your LLM Fully Utilize the Contextcited by 118
  4. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  5. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  6. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  7. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  8. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
  9. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  10. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  11. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  12. 2025End-to-End Test-Time Training for Long Contextcited by 53
  13. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  14. 2024Foundation Models for Video Understanding: A Surveycited by 46
  15. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  16. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  17. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  18. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  19. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  20. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43
  21. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  22. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  23. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  24. 2024Autoregressive Models in Vision: A Surveycited by 39
  25. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
  26. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  27. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 34
  28. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  29. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  30. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  31. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 31
  32. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  33. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  34. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  35. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 30
  36. 2025Astra: General Interactive World Model with Autoregressive Denoisingcited by 30
  37. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  38. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 25

50 most cited of 196 All 196 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  2. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
  3. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding
  4. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  5. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  6. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  7. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
  8. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input