mlsanityData to 7 Oct 2026

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

arXiv 2311.17005 ↗First posted 28 Nov 2023cs.CVcited by 1,086

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models. However, most benchmarks predominantly assess spatial understanding in the static image tasks, while overlooking temporal understanding in the dynamic video tasks. To alleviate this issue, we introduce a comprehensive Multi-modal Video understanding Benchmark, namely MVBench, which covers 20 challenging video tasks that cannot be effectively solved with a single frame. Specifically, we first introduce a novel static-to-dynamic method to define these temporal-related tasks. By transforming various static tasks into dynamic ones, we enable the systematic generation of video tasks that require a broad spectrum of temporal skills, ranging from perception to cognition. Then, guided by the task definition, we automatically convert public video annotations into multiple-choice QA to evaluate each task. On one hand, such a distinct paradigm allows us to build MVBench efficiently, without much manual intervention. On the other hand, it guarantees evaluation fairness with ground-truth video annotations, avoiding the biased scoring of LLMs. Moreover, we further develop a robust video MLLM baseline, i.e., VideoChat2, by progressive multi-modal training with diverse instruction-tuning data. The extensive results on our MVBench reveal that, the existing MLLMs are far from satisfactory in temporal understanding, while our VideoChat2 largely surpasses these leading models by over 15% on MVBench. All models and data are available at https://github.com/OpenGVLab/Ask-Anything.

Key facts. Cited by 1,086 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 83 of its 102 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

83 of 102 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 82

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  13. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  14. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  15. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  16. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  17. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  18. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  20. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  21. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  22. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  23. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  24. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  25. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  26. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  27. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  28. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  29. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  30. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  31. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  32. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  33. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  34. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  35. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  36. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  37. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  38. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 222; 2025: 471; 2026: 387.

1,086 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  6. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  7. 2025Qwen2.5-Omni Technical Reportcited by 704
  8. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  10. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  11. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
38 more
  1. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  2. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  3. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  4. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  5. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  7. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  8. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  10. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  11. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  12. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  13. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  14. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  15. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  16. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  17. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  18. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  19. 2023Video Understanding with Large Language Models: A Surveycited by 203
  20. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  21. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  22. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  23. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  24. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  25. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  26. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  27. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  28. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  29. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  30. 2026Qwen3.5-Omni Technical Reportcited by 146
  31. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  32. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  33. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  34. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 138
  35. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  36. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  37. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  38. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131

50 most cited of 1,086 All 1,086 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning
  2. 2025CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models
  3. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  4. 2026SYNCR: A Cross-Video Reasoning Benchmark with Synthetic Grounding
  5. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding
  6. 2026Training-Free Temporal Abstraction for General Video Understanding
  7. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  8. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos