mlsanityData to 7 Oct 2026

LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding

arXiv 2410.17434 ↗First posted 22 Oct 2024cs.CVcited by 341

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance.

Key facts. Cited by 341 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 67 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

56 of 67 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Long Context Transfer from Language to Vision24 Jun
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  3. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens4 Apr
  4. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  5. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
36 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2022Token Merging: Your ViT But Fastercited by 995
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  13. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  14. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  16. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  17. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  18. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  19. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  20. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  21. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  22. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  23. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  24. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  25. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  26. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  27. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  28. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  29. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  30. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  31. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  32. 2023Image as Set of Pointscited by 43
  33. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  34. 2024Goldfish: Vision-Language Understanding of Arbitrarily Long Videoscited by 38
  35. 2023A Large Cross-Modal Video Retrieval Dataset with Reading Comprehensioncited by 31
  36. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformerscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 152; 2026: 178.

341 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  3. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  4. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  5. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  6. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  7. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  8. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  9. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  10. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  11. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  12. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
38 more
  1. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  2. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  3. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  4. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  5. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  6. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  7. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 55
  8. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  9. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  10. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  11. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  12. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46
  13. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  14. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 46
  15. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  16. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  17. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  18. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  19. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
  20. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  21. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  22. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  23. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  24. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  25. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  26. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  27. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 30
  28. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  29. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  30. 2026A Simple Baseline for Streaming Video Understandingcited by 29
  31. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  32. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  33. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  34. 2025LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoningcited by 27
  35. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  36. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  37. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  38. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26

50 most cited of 341 All 341 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025FiLA-Video: Spatio-Temporal Compression for Fine-Grained Long Video Understanding
  2. 2025LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding
  3. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  4. 2024VidCompress: Memory-Enhanced Temporal Compression for Video Understanding in Large Language Models
  5. 2026Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention
  6. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  7. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models
  8. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding