mlsanityData to 7 Oct 2026

MM-VID: Advancing Video Understanding with GPT-4V(ision)

arXiv 2310.19773 ↗First posted 30 Oct 2023cs.CVcited by 82

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present MM-VID, an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding. MM-VID is designed to address the challenges posed by long-form videos and intricate tasks such as reasoning within hour-long content and grasping storylines spanning multiple episodes. MM-VID uses a video-to-script generation with GPT-4V to transcribe multimodal elements into a long textual script. The generated script details character movements, actions, expressions, and dialogues, paving the way for large language models (LLMs) to achieve video understanding. This enables advanced capabilities, including audio description, character identification, and multimodal high-level comprehension. Experimental results demonstrate the effectiveness of MM-VID in handling distinct video genres with various video lengths. Additionally, we showcase its potential when applied to interactive environments, such as video games and graphic user interfaces.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 61 of its 78 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

61 of 78 references.

References · 61

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  11. 2018SlowFast Networks for Video Recognitioncited by 1,862
  12. 2021ViViT: A Video Vision Transformercited by 1,610
38 more
  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  7. 2021Video Swin Transformercited by 1,090
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2021Multiscale Vision Transformerscited by 896
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  13. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  14. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  15. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  16. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  17. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  18. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  19. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  20. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  21. 2022Multimodal Learning with Transformers: A Surveycited by 399
  22. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  23. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  24. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  25. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  26. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  27. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  28. 2022Egocentric Video-Language Pretrainingcited by 231
  29. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  30. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  31. 2015A Dataset for Movie Descriptioncited by 195
  32. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  33. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  34. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  35. 2021Towards Long-Form Video Understandingcited by 146
  36. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  37. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  38. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 36; 2025: 29; 2026: 7.

82 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  3. 2023Video Understanding with Large Language Models: A Surveycited by 203
  4. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  5. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  6. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  7. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  8. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  9. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  10. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  11. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  12. 2024Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLMcited by 63
38 more
  1. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  2. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilitiescited by 57
  3. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  4. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  5. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  6. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  7. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  8. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
  9. 2024OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquercited by 34
  10. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 33
  11. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
  12. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 30
  13. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 24
  14. 2024Large Language Models and Video Games: A Preliminary Scoping Reviewcited by 23
  15. 2023How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigationcited by 22
  16. 2024AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Descriptioncited by 21
  17. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  18. 2025RAG-Anything: All-in-One RAG Frameworkcited by 18
  19. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  20. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  21. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answeringcited by 16
  22. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
  23. 2024Contextual AD Narration with Interleaved Multimodal Sequencecited by 14
  24. 2024iRAG: Advancing RAG for Videos with an Incremental Approachcited by 14
  25. 2024LLM-AD: Large Language Model based Audio Description Systemcited by 14
  26. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understandingcited by 14
  27. 2024MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoningcited by 13
  28. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  29. 2023Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Studycited by 12
  30. 2024DistinctAD: Distinctive Audio Description Generation in Contextscited by 11
  31. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  32. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8
  33. 2024LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Groundingcited by 7
  34. 2024Harnessing Business and Media Insights with Large Language Modelscited by 7
  35. 2024Details Make a Difference: Object State-Sensitive Neurorobotic Task Planningcited by 7
  36. 2024HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understandingcited by 7
  37. 2023GPT-4V(ision) as A Social Media Analysis Enginecited by 6
  38. 2024Zero-Shot Long-Form Video Understanding through Screenplaycited by 6

50 most cited of 82 All 82 (JSON)