# Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding arXiv: https://arxiv.org/abs/2507.15028 ยท HTML: https://mlsanity.com/paper/2507.15028/ ## Key facts - Cited by 24 papers in the map (data through 14 Oct 2026). - Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. - Builds on 17 of its 54 references that are in the map. - First posted on arXiv 20 Jul 2025. ## Placed in - LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/) ## Builds on (most cited first, 17 of 17) - [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024) - [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023) - [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024) - [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023) - [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024) - [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019) - [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018) - [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024) - [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024) - [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024) - [Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution](https://mlsanity.com/paper/2409.12961/) (2024) - [Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2411.14432/) (2024) - [TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models](https://mlsanity.com/paper/2410.10818/) (2024) - [Ola: Pushing the Frontiers of Omni-Modal Language Model](https://mlsanity.com/paper/2502.04328/) (2025) - [Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models](https://mlsanity.com/paper/2403.12966/) (2024) - [Visual Agents as Fast and Slow Thinkers](https://mlsanity.com/paper/2408.08862/) (2024) - [FunQA: Towards Surprising Video Comprehension](https://mlsanity.com/paper/2306.14899/) (2023) ## Built on by (most cited first, 24 of 24) - [Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding](https://mlsanity.com/paper/2604.05015/) (2026) - [AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video](https://mlsanity.com/paper/2508.03100/) (2025) - [CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling](https://mlsanity.com/paper/2602.13191/) (2026) - [Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models](https://mlsanity.com/paper/2510.05034/) (2025) - [Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition](https://mlsanity.com/paper/2602.08439/) (2026) - [VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification](https://mlsanity.com/paper/2604.01569/) (2026) - [GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinking](https://mlsanity.com/paper/2602.17555/) (2026) - [LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks](https://mlsanity.com/paper/2603.00490/) (2026) - [PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs](https://mlsanity.com/paper/2512.14944/) (2025) - [AVIS: Adaptive Test-Time Scaling for Vision-Language Models](https://mlsanity.com/paper/2606.11576/) (2026) - [Video Reasoning without Training](https://mlsanity.com/paper/2510.17045/) (2025) - [Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs](https://mlsanity.com/paper/2605.13737/) (2026) - [OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering](https://mlsanity.com/paper/2604.08209/) (2026) - [Decoupling Perception from Reasoning for Hallucination-Resistant Video Understanding](https://mlsanity.com/paper/2511.18463/) (2025) - [CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation](https://mlsanity.com/paper/2601.08010/) (2026) - [TGV-KV: Text-Grounded KV Eviction for Vision-Language Models](https://mlsanity.com/paper/2606.03075/) (2026) - [VideoLatent: Video-Language Learning via Latent Self-Forcing](https://mlsanity.com/paper/2606.22870/) (2026) - [Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering](https://mlsanity.com/paper/2608.04124/) (2026) - [MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents](https://mlsanity.com/paper/2608.07533/) (2026) - [Video-FLAIR: Not Whether to Reason, But How](https://mlsanity.com/paper/2608.26495/) (2026) - [TempCloze: Can Video-LLMs Identify the Missing Middle?](https://mlsanity.com/paper/2609.01515/) (2026) - [Agentic Video Understanding: A Survey](https://mlsanity.com/paper/2609.31713/) (2026) - [SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models](https://mlsanity.com/paper/2609.34044/) (2026) - [RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling](https://mlsanity.com/paper/2609.39563/) (2026) ## Abstract (arXiv) Human intelligence requires correctness and robustness, with the former being foundational for the latter. In video understanding, correctness ensures the accurate interpretation of visual content, and robustness maintains consistent performance in challenging conditions. Despite advances in video large language models (video LLMs), existing benchmarks inadequately reflect the gap between these models and human intelligence in maintaining correctness and robustness in video interpretation. We introduce the Video Thinking Test (Video-TT), to assess if video LLMs can interpret real-world videos as effectively as humans. Video-TT reflects genuine gaps in understanding complex visual narratives, and evaluates robustness against natural adversarial questions. Video-TT comprises 1,000 YouTube Shorts videos, each with one open-ended question and four adversarial questions that probe visual and narrative complexity. Our evaluation shows a significant gap between video LLMs and human performance.