mlsanityData to 14 Oct 2026

Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

arXiv 2405.21075 ↗First posted 31 May 2024cs.CV cs.CLcited by 1,413

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing benchmarks through four key features: 1) Diversity in video types, spanning 6 primary visual domains with 30 subfields to ensure broad scenario generalizability; 2) Duration in temporal dimension, encompassing both short-, medium-, and long-term videos, ranging from 11 seconds to 1 hour, for robust contextual dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides video frames, including subtitles and audios, to unveil the all-round capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual labeling by expert annotators to facilitate precise and reliable model assessment. 900 videos with a total of 254 hours are manually selected and annotated by repeatedly viewing all the video content, resulting in 2,700 question-answer pairs. With Video-MME, we extensively evaluate various state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as open-source image models like InternVL-Chat-V1.5 and video models like LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models. Our dataset along with these findings underscores the need for further improvements in handling longer sequences and multi-modal data. Project Page: https://video-mme.github.io

Key facts. Cited by 1,413 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 85 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

64 of 85 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  2. 2024Pegasus-v1 Technical Report23 Apr
  3. 2024LITA: Language Instructed Temporal-Localization Assistant27 Mar
  4. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  6. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering25 Nov
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2023A Survey on Multimodal Large Language Modelscited by 909
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  12. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  13. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  15. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  16. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  17. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  18. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  19. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  20. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  21. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  22. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  23. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  24. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  25. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  26. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  27. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  28. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  29. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  30. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  31. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  32. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  33. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  34. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  35. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  36. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 86
  37. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  38. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 145; 2025: 609; 2026: 659.

1,413 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2025Qwen3-VL Technical Reportcited by 2,793
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  9. 2025Qwen2.5-Omni Technical Reportcited by 736
  10. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  11. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  12. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
38 more
  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  3. 2025Qwen3-Omni Technical Reportcited by 497
  4. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  5. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  7. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  8. 2025Kimi-VL Technical Reportcited by 388
  9. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  10. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  11. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  12. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  13. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  14. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  15. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
  16. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  17. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  18. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  19. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  20. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  21. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  22. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  23. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  24. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
  25. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  26. 2026Qwen3.5-Omni Technical Reportcited by 174
  27. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  28. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
  29. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  30. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  31. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  32. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 159
  33. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  34. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  35. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  36. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  37. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  38. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140

50 most cited of 1,413 All 1,413 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  2. 2023VLM-Eval: A General Evaluation on Video Large Language Models
  3. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos
  4. 2025MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
  5. 2024From Image to Video, what do we need in multimodal LLMs?
  6. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs
  7. 2024A Review of Multi-Modal Large Language and Vision Models
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks