mlsanityData to 14 Oct 2026

MMVU: Measuring Expert-Level Multi-Discipline Video Understanding

arXiv 2501.12380 ↗First posted 21 Jan 2025cs.CV cs.AI cs.CLcited by 137

Not on the map. Why?

Cited by 58 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark for evaluating foundation models in video understanding. MMVU includes 3,000 expert-annotated questions spanning 27 subjects across four core disciplines: Science, Healthcare, Humanities & Social Sciences, and Engineering. Compared to prior benchmarks, MMVU features three key advancements. First, it challenges models to apply domain-specific knowledge and perform expert-level reasoning to analyze specialized-domain videos, moving beyond the basic visual perception typically assessed in current video benchmarks. Second, each example is annotated by human experts from scratch. We implement strict data quality controls to ensure the high quality of the dataset. Finally, each example is enriched with expert-annotated reasoning rationals and relevant domain knowledge, facilitating in-depth analysis. We conduct an extensive evaluation of 32 frontier multimodal foundation models on MMVU. The latest System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest performance among the tested models. However, they still fall short of matching human expertise. Through in-depth error analyses and case studies, we offer actionable insights for future advancements in expert-level, knowledge-intensive video understanding for specialized domains.

Key facts. Cited by 137 papers in the map (data through 14 Oct 2026). Builds on 59 of its 147 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

59 of 147 references.

References · 59

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
38 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  6. 2017Dense-Captioning Events in Videoscited by 793
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  8. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  11. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  12. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  13. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  14. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  15. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  16. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  17. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  18. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  19. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  20. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  21. 2024What matters when building vision-language models?cited by 326
  22. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  23. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  24. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  25. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  26. 2023Video Understanding with Large Language Models: A Surveycited by 206
  27. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  28. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  29. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  30. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  31. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 115
  32. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  33. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 98
  34. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  35. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 73
  36. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  37. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Modelscited by 41
  38. 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domainscited by 40

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 62; 2026: 75.

137 papers.

Sort:

By area:

  1. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  2. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  3. 2025Kimi-VL Technical Reportcited by 388
  4. 2026Qwen3.5-Omni Technical Reportcited by 174
  5. 2026Kimi K3: Open Frontier Intelligencecited by 138
  6. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 118
  7. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 93
  8. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  9. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  10. 2025TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoningcited by 62
  11. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  12. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
38 more
  1. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 55
  2. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 55
  3. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  4. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  5. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 47
  6. 2025Kwai Keye-VL Technical Reportcited by 47
  7. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 46
  8. 2026MiniMax Sparse Attentioncited by 44
  9. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 43
  10. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 42
  11. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  12. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
  13. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  14. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 33
  15. 2025Physics: Benchmarking Foundation Models on University-Level Physics Problem Solvingcited by 31
  16. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 31
  17. 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 30
  18. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 26
  19. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videoscited by 25
  20. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  21. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  22. 2025MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree Searchcited by 21
  23. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 21
  24. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 21
  25. 2025VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?cited by 20
  26. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 17
  27. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 16
  28. 2025SiLVR: A Simple Language-based Video Reasoning Frameworkcited by 14
  29. 2025VidBridge-R1: Bridging QA and Captioning for RL-based Video Understanding Models with Intermediate Proxy Taskscited by 14
  30. 2025MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Applicationcited by 14
  31. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  32. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 13
  33. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 12
  34. 2025Video SimpleQA: Towards Factuality Evaluation in Large Video Language Modelscited by 11
  35. 2026ERNIE 5.0 Technical Reportcited by 11
  36. 2025Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Paperscited by 10
  37. 2025RefineShot: Rethinking Cinematography Understanding with Foundational Skill Evaluationcited by 10
  38. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10

50 most cited of 137 All 137 (JSON)