mlsanityData to 7 Oct 2026

MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding

arXiv 2406.14515 ↗First posted 20 Jun 2024cs.CV cs.MMcited by 180

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

The advent of large vision-language models (LVLMs) has spurred research into their applications in multi-modal contexts, particularly in video understanding. Traditional VideoQA benchmarks, despite providing quantitative metrics, often fail to encompass the full spectrum of video content and inadequately assess models' temporal comprehension. To address these limitations, we introduce MMBench-Video, a quantitative benchmark designed to rigorously evaluate LVLMs' proficiency in video understanding. MMBench-Video incorporates lengthy videos from YouTube and employs free-form questions, mirroring practical use cases. The benchmark is meticulously crafted to probe the models' temporal reasoning skills, with all questions human-annotated according to a carefully constructed ability taxonomy. We employ GPT-4 for automated assessment, demonstrating superior accuracy and robustness over earlier LLM-based evaluations. Utilizing MMBench-Video, we have conducted comprehensive evaluations that include both proprietary and open-source LVLMs for images and videos. MMBench-Video stands as a valuable resource for the research community, facilitating improved evaluation of LVLMs and catalyzing progress in the field of video understanding. The evalutation code of MMBench-Video will be integrated into VLMEvalKit: https://github.com/open-compass/VLMEvalKit.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 77 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

39 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  2. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens4 Apr
  3. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
22 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  6. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  7. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  8. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  9. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  10. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  11. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  12. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  13. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  14. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  15. 2017Temporal Action Detection with Structured Segment Networkscited by 265
  16. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  17. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  18. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  19. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  20. 2019KnowIT VQA: Answering Knowledge-Based Questions about Videoscited by 53
  21. 2020Omni-sourced Webly-supervised Learning for Video Recognitioncited by 48
  22. 2016MarioQA: Answering Questions by Watching Gameplay Videoscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 99; 2026: 52.

180 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  5. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  6. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  7. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  10. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  11. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  12. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
38 more
  1. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  2. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  3. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  4. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  5. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  6. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  7. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  8. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  9. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  10. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 74
  11. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  12. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  13. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  14. 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularitycited by 66
  15. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  16. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 64
  17. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 55
  18. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  19. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  20. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  21. 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 48
  22. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  23. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 46
  24. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  25. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  26. 2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
  27. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  28. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  29. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  30. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  31. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  32. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  33. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 33
  34. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  35. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  36. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  37. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 24
  38. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1cited by 24

50 most cited of 180 All 180 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
  2. 2026MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding
  3. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  5. 2025HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding
  6. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?
  7. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
  8. 2026Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning