# MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos

arXiv: https://arxiv.org/abs/2506.04141 · HTML: https://mlsanity.com/paper/2506.04141/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 63 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos](https://mlsanity.com/paper/2501.13826/) (2025)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI](https://mlsanity.com/paper/2404.16006/) (2024)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions](https://mlsanity.com/paper/2411.14405/) (2024)
- [Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models](https://mlsanity.com/paper/2406.17294/) (2024)
- [CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding](https://mlsanity.com/paper/2412.12075/) (2024)
- [How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs](https://mlsanity.com/paper/2405.03690/) (2024)
- [HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks](https://mlsanity.com/paper/2410.12381/) (2024)

## Built on by (most cited first, 22 of 22)

- [Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning](https://mlsanity.com/paper/2601.23224/) (2026)
- [VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twice](https://mlsanity.com/paper/2601.05175/) (2026)
- [Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence](https://mlsanity.com/paper/2510.20470/) (2025)
- [JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence](https://mlsanity.com/paper/2606.14777/) (2026)
- [SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models](https://mlsanity.com/paper/2510.08559/) (2025)
- [Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences](https://mlsanity.com/paper/2510.23451/) (2025)
- [CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videos](https://mlsanity.com/paper/2507.16878/) (2025)
- [MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning](https://mlsanity.com/paper/2603.02024/) (2026)
- [LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understanding](https://mlsanity.com/paper/2601.15016/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification](https://mlsanity.com/paper/2604.01569/) (2026)
- [VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2511.19524/) (2025)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [Video-Oasis: Rethinking Evaluation of Video Understanding](https://mlsanity.com/paper/2603.29616/) (2026)
- [HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes](https://mlsanity.com/paper/2508.13692/) (2025)
- [SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs](https://mlsanity.com/paper/2608.03573/) (2026)
- [Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learning](https://mlsanity.com/paper/2605.21988/) (2026)
- [Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos](https://mlsanity.com/paper/2608.26355/) (2026)
- [MetaphorVU: Towards Metaphorical Video Understanding](https://mlsanity.com/paper/2605.25461/) (2026)
- [VideoLatent: Video-Language Learning via Latent Self-Forcing](https://mlsanity.com/paper/2606.22870/) (2026)
- [Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Models](https://mlsanity.com/paper/2609.25773/) (2026)
- [VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding](https://mlsanity.com/paper/2609.38413/) (2026)

## Abstract (arXiv)

The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly focus on understanding tasks, which only require models to match frames mentioned in the question (hereafter referred to as "question frame") and perceive a few adjacent frames. To address this gap, we propose MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos. The benchmark is characterized by the following features. (1) Long-range, multi-frame reasoning: Models are required to infer and analyze evidence frames that may be far from the question frame. (2) Beyond perception: Questions cannot be answered through direct perception alone but require reasoning over hidden information. (3) Reliability: All tasks are manually annotated, referencing extensive real-world user understanding to align with common perceptions. (4) Confusability: Carefully designed distractor annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos and 1,257 tasks. Our experiments reveal that current models still struggle with multi-modal reasoning; even the best-performing model, Gemini-2.5-pro, achieves only 64.3% accuracy. Additionally, current reasoning enhancement strategies (Chain-of-Thought and scaling test-time compute) bring limited gains. Error analysis indicates that the CoT demanded for multi-modal reasoning differs from it in textual reasoning, which partly explains the limited performance gains. We hope that MMR-V can inspire further research into enhancing multi-modal reasoning capabilities.