mlsanityData to 7 Oct 2026

MLVU: Benchmarking Multi-task Long Video Understanding

arXiv 2406.04264 ↗First posted 6 Jun 2024cs.CV cs.AI cs.CLcited by 275

Not on the map. Why?

Cited by 137 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The evaluation of Long Video Understanding (LVU) performance poses an important but challenging research problem. Despite previous efforts, the existing video understanding benchmarks are severely constrained by several issues, especially the insufficient lengths of videos, a lack of diversity in video types and evaluation tasks, and the inappropriateness for evaluating LVU performances. To address the above problems, we propose a new benchmark called MLVU (Multi-task Long Video Understanding Benchmark) for the comprehensive and in-depth evaluation of LVU. MLVU presents the following critical values: textit1) The substantial and flexible extension of video lengths, which enables the benchmark to evaluate LVU performance across a wide range of durations. textit2) The inclusion of various video genres, e.g., movies, surveillance footage, egocentric videos, cartoons, game videos, etc., which reflects the models' LVU performances in different scenarios. textit3) The development of diversified evaluation tasks, which enables a comprehensive examination of MLLMs' key abilities in long-video understanding. The empirical study with 23 latest MLLMs reveals significant room for improvement in today's technique, as all existing methods struggle with most of the evaluation tasks and exhibit severe performance degradation when handling longer videos. Additionally, it suggests that factors such as context length, image-understanding ability, and the choice of LLM backbone can play critical roles in future advancements. We anticipate that MLVU will advance the research of long video understanding by providing a comprehensive and in-depth analysis of MLLMs.

Key facts. Cited by 275 papers in the map (data through 7 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

49 of 65 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
37 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  12. 2017Dense-Captioning Events in Videoscited by 789
  13. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  14. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  15. 2018Real-world Anomaly Detection in Surveillance Videoscited by 570
  16. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  17. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  18. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  19. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  20. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  21. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  22. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  23. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  24. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  25. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  26. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  27. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  28. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  29. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  30. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  31. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  32. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  33. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  34. 2023FunQA: Towards Surprising Video Comprehensioncited by 35
  35. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  36. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  37. 2023Movie101: A New Movie Understanding Benchmarkcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 79; 2026: 173.

275 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  3. 2025Qwen3-Omni Technical Reportcited by 451
  4. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  5. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  6. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  7. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  8. 2026Qwen3.5-Omni Technical Reportcited by 146
  9. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  10. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  11. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  12. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
38 more
  1. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  2. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  3. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  5. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  6. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  7. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  8. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  9. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  10. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  11. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  12. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  13. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  14. 2024Temporal Reasoning Transfer from Text to Videocited by 36
  15. 2026MiniMax Sparse Attentioncited by 34
  16. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 30
  17. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 30
  18. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  19. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  20. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  21. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 29
  22. 2026A Simple Baseline for Streaming Video Understandingcited by 29
  23. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  24. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  25. 2024SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Modelscited by 24
  26. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  27. 2025Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understandingcited by 24
  28. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 24
  29. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videoscited by 23
  30. 2024STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Trainingcited by 22
  31. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videoscited by 22
  32. 2025V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Modelscited by 22
  33. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  34. 2025STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMscited by 21
  35. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 21
  36. 2025FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learningcited by 20
  37. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  38. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 17

50 most cited of 275 All 275 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
  2. 2025H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understanding
  3. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  4. 2025ALLVB: All-in-One Long Video Understanding Benchmark
  5. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video
  6. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding
  7. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
  8. 2025ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding