mlsanityData to 7 Oct 2026

Video Understanding with Large Language Models: A Survey

arXiv 2312.17432 ↗First posted 29 Dec 2023cs.CV cs.CLcited by 203

Not on the map. Why?

Cited by 67 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly. Given the remarkable capabilities of large language models (LLMs) in language and multimodal tasks, this survey provides a detailed overview of recent advancements in video understanding that harness the power of LLMs (Vid-LLMs). The emergent capabilities of Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity (general, temporal, and spatiotemporal) reasoning combined with commonsense knowledge, suggesting a promising path for future video understanding. We examine the unique characteristics and capabilities of Vid-LLMs, categorizing the approaches into three main types: Video Analyzer x LLM, Video Embedder x LLM, and (Analyzer + Embedder) x LLM. Furthermore, we identify five sub-types based on the functions of LLMs in Vid-LLMs: LLM as Summarizer, LLM as Manager, LLM as Text Decoder, LLM as Regressor, and LLM as Hidden Layer. Furthermore, this survey presents a comprehensive study of the tasks, datasets, benchmarks, and evaluation methodologies for Vid-LLMs. Additionally, it explores the expansive applications of Vid-LLMs across various domains, highlighting their remarkable scalability and versatility in real-world video understanding challenges. Finally, it summarizes the limitations of existing Vid-LLMs and outlines directions for future research. For more information, readers are recommended to visit the repository at https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). Builds on 225 of its 442 references that are in the map. First posted on arXiv 29 Dec 2023.

Builds on

225 of 442 references.

References · 225

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2017Squeeze-and-Excitation Networkscited by 4,763
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2017Non-local Neural Networkscited by 2,290
  13. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  14. 2017The Kinetics Human Action Video Datasetcited by 1,894
  15. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  16. 2018SlowFast Networks for Video Recognitioncited by 1,862
  17. 2021ViViT: A Video Vision Transformercited by 1,610
  18. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  19. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  20. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  21. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  22. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  23. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  24. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  25. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  26. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  27. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  28. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  29. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  30. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  31. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  32. 2022Token Merging: Your ViT But Fastercited by 995
  33. 2021Scaling Vision Transformerscited by 993
  34. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  35. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  36. 2023A Survey on Multimodal Large Language Modelscited by 898
  37. 2021Multiscale Vision Transformerscited by 896
  38. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891

50 most cited of 225 All 225 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 47; 2025: 79; 2026: 77.

203 papers.

Sort:

By area:

  1. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  2. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  3. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  4. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  5. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  6. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  7. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 72
  8. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  9. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  10. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  11. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
  12. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 55
38 more
  1. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  2. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  3. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  4. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  5. 2024Graphic Design with Large Multimodal Modelcited by 38
  6. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  7. 2024Temporal Reasoning Transfer from Text to Videocited by 36
  8. 2024OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquercited by 34
  9. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 34
  10. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  11. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  12. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  13. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  14. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  15. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  16. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  17. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelitycited by 23
  18. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  19. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 21
  20. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  21. 2024Towards Event-oriented Long Video Understandingcited by 20
  22. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  23. 2024Automating the Search for Artificial Life with Foundation Modelscited by 20
  24. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 20
  25. 2025FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learningcited by 20
  26. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  27. 2025T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attackscited by 18
  28. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Modelscited by 18
  29. 2025Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platformscited by 18
  30. 2024Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMscited by 17
  31. 2025Generative AI for Cel-Animation: A Surveycited by 17
  32. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  33. 2025CreatiPoster: Towards Editable and Controllable Multi-Layer Graphic Design Generationcited by 17
  34. 2024Tri2-plane: Thinking Head Avatar via Feature Pyramidcited by 16
  35. 2024OSCaR: Object State Captioning and State Change Representationcited by 16
  36. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  37. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  38. 2024Video2BEV: Transforming Drone Videos to BEVs for Video-based Geo-localizationcited by 16

50 most cited of 203 All 203 (JSON)