mlsanityData to 7 Oct 2026

MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

arXiv 2404.03413 ↗First posted 4 Apr 2024cs.CVcited by 116

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM) designed specifically for video understanding. The model is capable of processing both temporal visual and textual data, making it adept at understanding the complexities of videos. Building upon the success of MiniGPT-v2, which excelled in translating visual features into the LLM space for single images and achieved impressive results on various image-text benchmarks, this paper extends the model's capabilities to process a sequence of frames, enabling it to comprehend videos. MiniGPT4-video does not only consider visual content but also incorporates textual conversations, allowing the model to effectively answer queries involving both visual and text components. The proposed model outperforms existing state-of-the-art methods, registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF, and TVQA benchmarks respectively. Our models and code have been made publicly available here https://vision-cair.github.io/MiniGPT4-video/

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 33 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

28 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023Mistral 7Bcited by 3,065
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
13 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  7. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  9. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  12. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  13. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 62; 2026: 25.

116 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  4. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  5. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  6. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  7. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  8. 2024An Introduction to Vision-Language Modelingcited by 175
  9. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  10. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  11. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  12. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
38 more
  1. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  2. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  3. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  4. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  5. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  6. 2024Animate3D: Animating Any 3D Model with Multi-view Video Diffusioncited by 55
  7. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 55
  8. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 52
  9. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  10. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  11. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  12. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  13. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  14. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 30
  15. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  16. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 27
  17. 2024MECD: Unlocking Multi-Event Causal Discovery in Video Reasoningcited by 23
  18. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  19. 2025LongVideoAgent: Multi-Agent Reasoning with Long Videoscited by 23
  20. 2024STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Trainingcited by 22
  21. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 22
  22. 2024VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMscited by 20
  23. 2024InfiniBench: A Benchmark for Large Multi-Modal Models in Long-Form Movies and TV Showscited by 20
  24. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  25. 2025Memory-enhanced Retrieval Augmentation for Long Video Understandingcited by 17
  26. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  27. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  28. 2024Do Language Models Understand Time?cited by 14
  29. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  30. 2024GPT-4o: Visual perception performance of multimodal large language models in piglet activity understandingcited by 12
  31. 2024VideoOrion: Tokenizing Object Dynamics in Videoscited by 12
  32. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 12
  33. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  34. 2025K-frames: Scene-Driven Any-k Keyframe Selection for long video understandingcited by 11
  35. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  36. 2024DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understandingcited by 10
  37. 2025LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Modelscited by 10
  38. 2025VidText: Towards Comprehensive Evaluation for Video Text Understandingcited by 10

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding
  2. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models
  3. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
  4. 2023VideoLLM: Modeling Video Sequence with Large Language Models
  5. 2023Audio-Visual LLM for Video Understanding
  6. 2024From Image to Video, what do we need in multimodal LLMs?
  7. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation
  8. 2024Understanding Long Videos with Multimodal Language Models