mlsanityData to 14 Oct 2026

Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

arXiv 2504.14693 ↗First posted 20 Apr 2025cs.CV cs.AIcited by 37

Not on the map. Why?

Abstract (arXiv)

Recent advancements in language multimodal models (LMMs) for video have demonstrated their potential for understanding video content, yet the task of comprehending multi-discipline lectures remains largely unexplored. We introduce Video-MMLU, a massive benchmark designed to evaluate the capabilities of LMMs in understanding Multi-Discipline Lectures. We evaluate over 90 open-source and proprietary models, ranging from 0.5B to 40B parameters. Our results highlight the limitations of current models in addressing the cognitive challenges presented by these lectures, especially in tasks requiring both perception and reasoning. Additionally, we explore how the number of visual tokens and the large language models influence performance, offering insights into the interplay between multimodal perception and reasoning in lecture comprehension.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 148 of its 200 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

148 of 200 references.

References · 148

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
38 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2021ViViT: A Video Vision Transformercited by 1,620
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  10. 2021Video Swin Transformercited by 1,091
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  13. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  14. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  15. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  16. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  17. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  18. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  19. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  20. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  21. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  22. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  23. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  24. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  25. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  26. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  27. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  28. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  29. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  30. 2024InternLM2 Technical Reportcited by 397
  31. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  32. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  33. 2024Measuring short-form factuality in large language modelscited by 373
  34. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  35. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  36. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  37. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  38. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326

50 most cited of 148 All 148 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 14.

37 papers.

Sort:

By area:

  1. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  3. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  4. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 41
  5. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 34
  6. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 27
  7. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videoscited by 25
  8. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  9. 2025TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Modelscited by 19
  10. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methodscited by 17
  11. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 14
  12. 2025SiLVR: A Simple Language-based Video Reasoning Frameworkcited by 14
25 more
  1. 2025SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Modelscited by 13
  2. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  3. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  4. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  5. 2025MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMscited by 10
  6. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  7. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  8. 2025ExAct: A Video-Language Benchmark for Expert Action Analysiscited by 8
  9. 2025AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyondcited by 8
  10. 2026Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisitioncited by 8
  11. 2025MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialoguescited by 7
  12. 2026VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verificationcited by 7
  13. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  14. 2026Can Vision-Language Models Solve the Shell Game?cited by 3
  15. 2026Can LLMs Perceive Time? An Empirical Investigationcited by 3
  16. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  17. 2026VEBench:Benchmarking Large Multimodal Models for Real-World Video Editingcited by 2
  18. 2026VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understandingcited by 2
  19. 2025M3-Verse: A "Spot the Difference" Challenge for Large Multimodal Modelscited by 1
  20. 2026RefereeBench: Are Video MLLMs Ready to be Multi-Sport Refereescited by 1
  21. 2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Modelscited by 1
  22. 2026OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environmentscited by 1
  23. 2026MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMscited by 1
  24. 2026MMHBench: A Multi-Perspective Benchmark for Mental Health Understanding in Long-Form Videoscited by 1
  25. 2026The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeepingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LEMON: How Well Do MLLMs Perform Temporal Multimodal Understanding on Instructional Videos?
  2. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
  3. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs
  4. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  5. 2026Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios
  6. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos
  7. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  8. 2026MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding