mlsanityData to 7 Oct 2026

Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding

arXiv 2409.14485 ↗First posted 22 Sep 2024cs.CVcited by 210

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Long video understanding poses a significant challenge for current Multi-modal Large Language Models (MLLMs). Notably, the MLLMs are constrained by their limited context lengths and the substantial costs while processing long videos. Although several existing methods attempt to reduce visual tokens, their strategies encounter severe bottleneck, restricting MLLMs' ability to perceive fine-grained visual details. In this work, we propose Video-XL, a novel approach that leverages MLLMs' inherent key-value (KV) sparsification capacity to condense the visual input. Specifically, we introduce a new special token, the Visual Summarization Token (VST), for each interval of the video, which summarizes the visual information within the interval as its associated KV. The VST module is trained by instruction fine-tuning, where two optimizing strategies are offered. 1.Curriculum learning, where VST learns to make small (easy) and large compression (hard) progressively. 2. Composite data curation, which integrates single-image, multi-image, and synthetic data to overcome the scarcity of long-video instruction data. The compression quality is further improved by dynamic compression, which customizes compression granularity based on the information density of different video intervals. Video-XL's effectiveness is verified from three aspects. First, it achieves a superior long-video understanding capability, outperforming state-of-the-art models of comparable sizes across multiple popular benchmarks. Second, it effectively preserves video information, with minimal compression loss even at 16x compression ratio. Third, it realizes outstanding cost-effectiveness, enabling high-quality processing of thousands of frames on a single A100 GPU.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 64 references that are in the map. First posted on arXiv 22 Sep 2024.

Builds on

50 of 64 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  2. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos26 Aug
  3. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  4. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  5. 2024Long Context Transfer from Language to Vision24 Jun
  6. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoning17 Jun
  7. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding13 Jun
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  9. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  10. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding8 Apr
  11. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens4 Apr
  12. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
4 more
  1. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
22 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  6. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  7. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  11. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  12. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  13. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  14. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  15. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  16. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  17. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  18. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  19. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 120
  20. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  21. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  22. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 78

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 100; 2026: 95.

210 papers.

Sort:

By area:

  1. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  2. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  3. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  4. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  5. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  6. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  7. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  8. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  9. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  10. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 74
  11. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  12. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 72
38 more
  1. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  2. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  3. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  4. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  5. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  6. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  7. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  8. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  9. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  10. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  11. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  12. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  13. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  14. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 40
  15. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  16. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  17. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  18. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  19. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  20. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  21. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  22. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  23. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  24. 2025VideoExplorer: Think With Videos For Agentic Long-Video Understandingcited by 29
  25. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  26. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  27. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  28. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  29. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  30. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  31. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  32. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 25
  33. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 24
  34. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  35. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  36. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  37. 2024Towards Universal Soccer Video Understandingcited by 21
  38. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluationcited by 21

50 most cited of 210 All 210 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  2. 2025FiLA-Video: Spatio-Temporal Compression for Fine-Grained Long Video Understanding
  3. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  4. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding
  5. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
  6. 2024Streaming Long Video Understanding with Large Language Models
  7. 2025Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding
  8. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding