mlsanityData to 14 Oct 2026

Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?

arXiv 2505.21374 ↗First posted 27 May 2025cs.CVcited by 118

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Recent advances in CoT reasoning and RL post-training have been reported to enhance video reasoning capabilities of MLLMs. This progress naturally raises a question: can these models perform complex video reasoning in a manner comparable to human experts? However, existing video benchmarks primarily evaluate visual perception and grounding abilities, with questions that can be answered based on explicit prompts or isolated visual cues. Such benchmarks do not fully capture the intricacies of real-world reasoning, where humans must actively search for, integrate, and analyze multiple clues before reaching a conclusion. To address this issue, we present Video-Holmes, a benchmark inspired by the reasoning process of Sherlock Holmes, designed to evaluate the complex video reasoning capabilities of MLLMs. Video-Holmes consists of 1,837 questions derived from 270 manually annotated suspense short films, which spans seven carefully designed tasks. Each task is constructed by first identifying key events and causal relationships within films, and then designing questions that require models to actively locate and connect multiple relevant visual clues scattered across different video segments. Our comprehensive evaluation of state-of-the-art MLLMs reveals that, while these models generally excel at visual perception, they encounter substantial difficulties with integrating information and often miss critical clues. For example, the best-performing model, Gemini-2.5-Pro, achieves an accuracy of only 45%, with most models scoring below 40%. We aim that Video-Holmes can serve as a "Holmes-test" for multimodal reasoning, motivating models to reason more like humans and emphasizing the ongoing challenges in this field. The benchmark is released in https://github.com/TencentARC/Video-Holmes.

Key facts. Cited by 118 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 43 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

35 of 43 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoning13 Apr
  2. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning10 Apr
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  4. 2025Kimi-VL Technical Report10 Apr
  5. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning9 Apr
  6. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R131 Mar
  7. 2025Video-R1: Reinforcing Video Reasoning in MLLMs27 Mar
  8. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning14 Mar
  9. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs10 Jan
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  11. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov

Other references · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  12. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
12 more
  1. 2025Qwen2.5-Omni Technical Reportcited by 736
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  5. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  6. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  7. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 236
  8. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  9. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  10. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  11. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  12. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 35; 2026: 83.

118 papers.

Sort:

By area:

  1. 2025Qwen3-Omni Technical Reportcited by 497
  2. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  3. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
  4. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  5. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  6. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  7. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  8. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 42
  9. 2025ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shortscited by 41
  10. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 34
  11. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 33
  12. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 30
38 more
  1. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 29
  2. 2025ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesiscited by 26
  3. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 26
  4. 2026Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoningcited by 24
  5. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 22
  6. 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 21
  7. 2025Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidencecited by 18
  8. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 18
  9. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 17
  10. 2025VidBridge-R1: Bridging QA and Captioning for RL-based Video Understanding Models with Intermediate Proxy Taskscited by 14
  11. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  12. 2025SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Modelscited by 13
  13. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understandingcited by 13
  14. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 12
  15. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  16. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 11
  17. 2025ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understandingcited by 10
  18. 2026Video-KTR: Reinforcing Video Reasoning via Key Token Attributioncited by 10
  19. 2025CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videoscited by 9
  20. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  21. 2025Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPOcited by 9
  22. 2025Rethinking Chain-of-Thought Reasoning for Videoscited by 9
  23. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  24. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  25. 2026VideoBrain: Learning Adaptive Frame Sampling for Long Video Understandingcited by 7
  26. 2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
  27. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  28. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  29. 2026EVA: Efficient Reinforcement Learning for End-to-End Video Agentcited by 6
  30. 2025DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learningcited by 5
  31. 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Modelscited by 5
  32. 2025Boosting Reasoning in Large Multimodal Models via Activation Replaycited by 5
  33. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMscited by 5
  34. 2025A Benchmark for Omni-Modal Reasoning in Long Videoscited by 5
  35. 2026Co-Evolving Policy Distillationcited by 5
  36. 2026StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggeringcited by 5
  37. 2026VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimizationcited by 5
  38. 2026VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Promptingcited by 4

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?
  2. 2025Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding
  3. 2024Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events
  4. 2025MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning
  5. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding
  6. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos
  7. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video
  8. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning