# MMVU: Measuring Expert-Level Multi-Discipline Video Understanding

arXiv: https://arxiv.org/abs/2501.12380 · HTML: https://mlsanity.com/paper/2501.12380/

## Key facts

- Cited by 137 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 147 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Builds on (most cited first, 30 of 59)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)

## Built on by (most cited first, 30 of 137)

- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?](https://mlsanity.com/paper/2505.21374/) (2025)
- [VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning](https://mlsanity.com/paper/2505.12434/) (2025)
- [Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning](https://mlsanity.com/paper/2508.04416/) (2025)
- [Kwai Keye-VL 1.5 Technical Report](https://mlsanity.com/paper/2509.01563/) (2025)
- [TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoning](https://mlsanity.com/paper/2504.09641/) (2025)
- [DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO](https://mlsanity.com/paper/2506.07464/) (2025)
- [OneThinker: All-in-one Reasoning Model for Image and Video](https://mlsanity.com/paper/2512.03043/) (2025)
- [HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context](https://mlsanity.com/paper/2506.21277/) (2025)
- [Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding](https://mlsanity.com/paper/2604.05015/) (2026)
- [ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding](https://mlsanity.com/paper/2506.01300/) (2025)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning](https://mlsanity.com/paper/2506.16141/) (2025)
- [Kwai Keye-VL Technical Report](https://mlsanity.com/paper/2507.01949/) (2025)
- [VLM4D: Towards Spatiotemporal Awareness in Vision Language Models](https://mlsanity.com/paper/2508.02095/) (2025)
- [MiniMax Sparse Attention](https://mlsanity.com/paper/2606.13392/) (2026)
- [Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark](https://mlsanity.com/paper/2510.26802/) (2025)
- [VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning](https://mlsanity.com/paper/2504.07956/) (2025)
- [Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark](https://mlsanity.com/paper/2504.14693/) (2025)
- [LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model](https://mlsanity.com/paper/2509.00676/) (2025)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoning](https://mlsanity.com/paper/2507.06485/) (2025)
- [Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving](https://mlsanity.com/paper/2503.21821/) (2025)
- [Reinforcing Video Reasoning with Focused Thinking](https://mlsanity.com/paper/2505.24718/) (2025)
- [VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking](https://mlsanity.com/paper/2506.01725/) (2025)
- [Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency](https://mlsanity.com/paper/2506.01908/) (2025)

## Abstract (arXiv)

We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark for evaluating foundation models in video understanding. MMVU includes 3,000 expert-annotated questions spanning 27 subjects across four core disciplines: Science, Healthcare, Humanities & Social Sciences, and Engineering. Compared to prior benchmarks, MMVU features three key advancements. First, it challenges models to apply domain-specific knowledge and perform expert-level reasoning to analyze specialized-domain videos, moving beyond the basic visual perception typically assessed in current video benchmarks. Second, each example is annotated by human experts from scratch. We implement strict data quality controls to ensure the high quality of the dataset. Finally, each example is enriched with expert-annotated reasoning rationals and relevant domain knowledge, facilitating in-depth analysis. We conduct an extensive evaluation of 32 frontier multimodal foundation models on MMVU. The latest System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest performance among the tested models. However, they still fall short of matching human expertise. Through in-depth error analyses and case studies, we offer actionable insights for future advancements in expert-level, knowledge-intensive video understanding for specialized domains.