mlsanityData to 14 Oct 2026

LongVILA: Scaling Long-Context Visual Language Models for Long Videos

arXiv 2408.10188 ↗First posted 19 Aug 2024cs.CV cs.CLcited by 309

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Long-context capability is critical for multi-modal foundation models, especially for long video understanding. We introduce LongVILA, a full-stack solution for long-context visual-language models by co-designing the algorithm and system. For model training, we upgrade existing VLMs to support long video understanding by incorporating two additional stages, i.e., long context extension and long video supervised fine-tuning. However, training on long video is computationally and memory intensive. We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradient checkpointing. LongVILA efficiently extends the number of video frames of VILA from 8 to 2048, achieving 99.8% accuracy in 6,000-frame (more than 1 million tokens) video needle-in-a-haystack. LongVILA-7B demonstrates strong accuracy on 9 popular video benchmarks, e.g. 65.1% VideoMME with subtitle. Besides, MM-SP is 2.1x - 5.7x faster than ring style sequence parallelism and 1.1x - 1.4x faster than Megatron with a hybrid context and tensor parallelism. Moreover, it seamlessly integrates with Hugging Face Transformers.

Key facts. Cited by 309 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 75 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

57 of 75 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  2. 2024Long Context Transfer from Language to Vision24 Jun
  3. 2024Towards Event-oriented Long Video Understanding20 Jun
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  5. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  8. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  11. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  12. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  13. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  14. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  15. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  16. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  17. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  18. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  19. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  20. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  21. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  22. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  23. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  24. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  25. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  26. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  27. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  28. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  29. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  30. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  31. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  32. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  33. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  34. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 76
  35. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 61
  36. 2023Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videoscited by 54
  37. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  38. 2024LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelismcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 155; 2026: 122.

309 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  3. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  4. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  5. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  6. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  7. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  8. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  9. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  10. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  11. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
  12. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
38 more
  1. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  2. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  3. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 106
  4. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  5. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
  6. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 97
  7. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 96
  8. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 95
  9. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  10. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  11. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 85
  12. 2025Scaling RL to Long Videoscited by 84
  13. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 83
  14. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  15. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 80
  16. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  17. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  18. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
  19. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  20. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  21. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  22. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 58
  23. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 57
  24. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
  25. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 49
  26. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49
  27. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 48
  28. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  29. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 43
  30. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 42
  31. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  32. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  33. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  34. 2025Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Modelscited by 38
  35. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 37
  36. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  37. 2025Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoningcited by 37
  38. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 37

50 most cited of 309 All 309 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  2. 2025Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMs
  3. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding
  4. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
  5. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
  6. 2025Video Panels for Long Video Understanding
  7. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input
  8. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints