mlsanityData to 7 Oct 2026

Long Context Transfer from Language to Vision

arXiv 2406.16852 ↗First posted 24 Jun 2024cs.CVcited by 502

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Video sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, we approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon long context transfer and carefully ablate its properties. To effectively measure LMMs' ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark inspired by the language model's NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the-art performance on Video-MME among 7B-scale models by densely sampling more input frames. Our work is open-sourced at https://github.com/EvolvingLMMs-Lab/LongVA.

Key facts. Cited by 502 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 61 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

39 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
25 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2016A Diagram Is Worth A Dozen Imagescited by 793
  4. 2023Large Language Modelscited by 774
  5. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  13. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  14. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  15. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  16. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  17. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  18. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  19. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  20. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  21. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  22. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  23. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  24. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  25. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 61; 2025: 257; 2026: 184.

502 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  7. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  8. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  9. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  10. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  11. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  12. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
38 more
  1. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  2. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  3. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  4. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  5. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  6. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  7. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  8. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  9. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  10. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  11. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  12. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 123
  13. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  14. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  15. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  16. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  17. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  18. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  19. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  20. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  21. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  22. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  23. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  24. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  25. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  26. 2025Scaling RL to Long Videoscited by 83
  27. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  28. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 80
  29. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  30. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  31. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  32. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 72
  33. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  34. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  35. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  36. 2025Virgo: A Preliminary Exploration on Reproducing o1-like MLLMcited by 68
  37. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  38. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65

50 most cited of 502 All 502 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MMLongCite: A Benchmark for Evaluating Fidelity of Long-Context Vision-Language Models
  2. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly
  3. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  4. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
  5. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  6. 2024Understanding Long Videos with Multimodal Language Models
  7. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
  8. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token