mlsanityData to 7 Oct 2026

LongVLM: Efficient Long Video Understanding via Large Language Models

arXiv 2404.03384 ↗First posted 4 Apr 2024cs.CVcited by 146

Not on the map. Why?

Cited by 64 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Empowered by Large Language Models (LLMs), recent advancements in Video-based LLMs (VideoLLMs) have driven progress in various video understanding tasks. These models encode video representations through pooling or query aggregation over a vast number of visual tokens, making computational and memory costs affordable. Despite successfully providing an overall comprehension of video content, existing VideoLLMs still face challenges in achieving detailed understanding due to overlooking local information in long-term videos. To tackle this challenge, we introduce LongVLM, a simple yet powerful VideoLLM for long video understanding, building upon the observation that long videos often consist of sequential key events, complex actions, and camera movements. Our approach proposes to decompose long videos into multiple short-term segments and encode local features for each segment via a hierarchical token merging module. These features are concatenated in temporal order to maintain the storyline across sequential short-term segments. Additionally, we propose to integrate global semantics into each local feature to enhance context understanding. In this way, we encode video representations that incorporate both local and global information, enabling the LLM to generate comprehensive responses for long-term videos. Experimental results on the VideoChatGPT benchmark and zero-shot video question-answering datasets demonstrate the superior capabilities of our model over the previous state-of-the-art methods. Qualitative examples show that our model produces more precise responses for long video understanding. Code is available at https://github.com/ziplab/LongVLM.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Builds on 57 of its 69 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

57 of 69 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2022Token Merging: Your ViT But Fastercited by 995
  5. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  13. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  14. 2017Temporal Relational Reasoning in Videoscited by 324
  15. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  16. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  17. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  18. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  19. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  20. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  21. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  22. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  23. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  24. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  25. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  26. 2021Towards Long-Form Video Understandingcited by 146
  27. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  28. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 133
  29. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 107
  30. 2018Timeception for Complex Action Recognitioncited by 89
  31. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  32. 2022Temporal Alignment Networks for Long-term Videocited by 84
  33. 2020SMART Frame Selection for Action Recognitioncited by 73
  34. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  35. 2021Temporal Query Networks for Fine-grained Video Understandingcited by 68
  36. 2021FrameExit: Conditional Early Exiting for Efficient Video Recognitioncited by 59
  37. 2018Learning Dynamic Memory Networks for Object Trackingcited by 58
  38. 2023Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videoscited by 54

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 87; 2026: 35.

146 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  3. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  4. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  5. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  8. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  9. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  10. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  11. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  12. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
38 more
  1. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  2. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  3. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  4. 2025Scaling RL to Long Videoscited by 83
  5. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  6. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  7. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  8. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  9. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  10. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  11. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  12. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  13. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  14. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  15. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  16. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  17. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  18. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  19. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  20. 2025AToken: A Unified Tokenizer for Visioncited by 30
  21. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  22. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  23. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  24. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  25. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  26. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  27. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  28. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  29. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  30. 2024StreamChat: Chatting with Streaming Videocited by 23
  31. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  32. 2025MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representationscited by 22
  33. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  34. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 21
  35. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  36. 2025SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Modelscited by 21
  37. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 18
  38. 2025SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Modelscited by 18

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding
  2. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding
  3. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  4. 2026Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Models
  5. 2023VideoLLM: Modeling Video Sequence with Large Language Models
  6. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  7. 2025Video Panels for Long Video Understanding
  8. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding