mlsanityData to 7 Oct 2026

Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding

arXiv 2507.15028 ↗First posted 20 Jul 2025cs.CVcited by 24

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Human intelligence requires correctness and robustness, with the former being foundational for the latter. In video understanding, correctness ensures the accurate interpretation of visual content, and robustness maintains consistent performance in challenging conditions. Despite advances in video large language models (video LLMs), existing benchmarks inadequately reflect the gap between these models and human intelligence in maintaining correctness and robustness in video interpretation. We introduce the Video Thinking Test (Video-TT), to assess if video LLMs can interpret real-world videos as effectively as humans. Video-TT reflects genuine gaps in understanding complex visual narratives, and evaluates robustness against natural adversarial questions. Video-TT comprises 1,000 YouTube Shorts videos, each with one open-ended question and four adversarial questions that probe visual and narrative complexity. Our evaluation shows a significant gap between video LLMs and human performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 54 references that are in the map. First posted on arXiv 20 Jul 2025.

Builds on

17 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models21 Nov
  3. 2024Visual Agents as Fast and Slow Thinkers16 Aug
  4. 2024Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models19 Mar

Other references · 13

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  4. 2024Long Context Transfer from Language to Visioncited by 502
  5. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  6. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  7. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  8. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  9. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  10. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  11. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  12. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
1 more
  1. 2023FunQA: Towards Surprising Video Comprehensioncited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 19.

24 papers.

Sort:

By area:

  1. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 46
  2. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  3. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  4. 2026Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisitioncited by 8
  5. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  6. 2026VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verificationcited by 6
  7. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  8. 2025PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMscited by 4
  9. 2026LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Taskscited by 4
  10. 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Modelscited by 4
  11. 2025Video Reasoning without Trainingcited by 2
  12. 2026Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMscited by 2
12 more
  1. 2026TGV-KV: Text-Grounded KV Eviction for Vision-Language Modelscited by 1
  2. 2025Decoupling Perception from Reasoning for Hallucination-Resistant Video Understandingcited by 0
  3. 2026CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregationcited by 0
  4. 2026OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reorderingcited by 0
  5. 2026VideoLatent: Video-Language Learning via Latent Self-Forcingcited by 0
  6. 2026Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answeringcited by 0
  7. 2026MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agentscited by 0
  8. 2026Video-FLAIR: Not Whether to Reason, But Howcited by 0
  9. 2026TempCloze: Can Video-LLMs Identify the Missing Middle?cited by 0
  10. 2026Agentic Video Understanding: A Surveycited by 0
  11. 2026SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Modelscited by 0
  12. 2026RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos
  2. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling
  3. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?
  4. 2026Video-Oasis: Rethinking Evaluation of Video Understanding
  5. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMs
  6. 2025GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?
  7. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
  8. 2026A Very Big Video Reasoning Suite