mlsanityData to 14 Oct 2026

Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought

arXiv 2506.08817 ↗First posted 10 Jun 2025cs.CVcited by 14

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Video content comprehension is essential for various applications, ranging from video analysis to interactive systems. Despite advancements in large-scale vision-language models (VLMs), these models often struggle to capture the nuanced, spatiotemporal details essential for thorough video analysis. To address this gap, we introduce Video-CoT, a groundbreaking dataset designed to enhance spatiotemporal understanding using Chain-of-Thought (CoT) methodologies. Video-CoT contains 192,000 fine-grained spa-tiotemporal question-answer pairs and 23,000 high-quality CoT-annotated samples, providing a solid foundation for evaluating spatiotemporal understanding in video comprehension. Additionally, we provide a comprehensive benchmark for assessing these tasks, with each task featuring 750 images and tailored evaluation metrics. Our extensive experiments reveal that current VLMs face significant challenges in achieving satisfactory performance, high-lighting the difficulties of effective spatiotemporal understanding. Overall, the Video-CoT dataset and benchmark open new avenues for research in multimedia understanding and support future innovations in intelligent systems requiring advanced video analysis capabilities. By making these resources publicly available, we aim to encourage further exploration in this critical area. Project website:https://video-cot.github.io/ .

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 53 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

25 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoning13 Apr
  2. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning9 Apr
  3. 2025Video-R1: Reinforcing Video Reasoning in MLLMs27 Mar
  4. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Model8 Jan
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2017The Kinetics Human Action Video Datasetcited by 1,909
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  11. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  12. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
8 more
  1. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  2. 2023Audio-Visual LLM for Video Understandingcited by 77
  3. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  4. 2025MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigationcited by 60
  5. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  6. 2025AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Cluttercited by 20
  7. 2024Multi-Floor Zero-Shot Object Navigation Policycited by 12
  8. 2025A Hierarchical Reinforcement Learning Framework for Multi-UAV Combat Using Leader-Follower Strategycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2025ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesiscited by 26
  2. 2025Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigationcited by 10
  3. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  4. 2026MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hintscited by 5
  5. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  6. 2026TIR-Flow: Active Video Search and Reasoning with Frozen VLMscited by 3
  7. 2026Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligencecited by 3
  8. 2026SEF-MAP: Subspace-Decomposed Expert Fusion for Robust Multimodal HD Map Predictioncited by 2
  9. 2026VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understandingcited by 2
  10. 2026Weather-Conditioned Branch Routing for Robust LiDAR-Radar 3D Object Detectioncited by 1
  11. 2026OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reorderingcited by 1
  12. 2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Modelscited by 1
2 more
  1. 2026Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understandingcited by 0
  2. 2026Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks
  2. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition
  3. 2026VISTA: Video Interaction Spatio-Temporal Analysis Benchmark
  4. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
  5. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
  6. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models
  7. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning
  8. 2026Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence