mlsanityData to 7 Oct 2026

Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition

arXiv 2501.03230 ↗First posted 7 May 2024cs.AI cs.CVcited by 151

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Existing research of video understanding still struggles to achieve in-depth comprehension and reasoning in complex videos, primarily due to the under-exploration of two key bottlenecks: fine-grained spatial-temporal perceptive understanding and cognitive-level video scene comprehension. This paper bridges the gap by presenting a novel solution. We first introduce a novel video Multimodal Large Language Model (MLLM), MotionEpic, which achieves fine-grained pixel-level spatial-temporal video grounding by integrating video spatial-temporal scene graph (STSG) representation. Building upon MotionEpic, we then develop a Video-of-Thought (VoT) reasoning framework. VoT inherits the Chain-of-Thought (CoT) core, breaking down a complex task into simpler and manageable sub-problems, and addressing them step-by-step from a low-level pixel perception to high-level cognitive interpretation. Extensive experiments across various complex video QA benchmarks demonstrate that our overall framework strikingly boosts existing state-of-the-art. To our knowledge, this is the first attempt at successfully implementing the CoT technique for achieving human-level video reasoning, where we show great potential in extending it to a wider range of video understanding scenarios. Project is open at https://haofei.vip/VoT

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 56 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

41 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answering24 Oct
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  4. 2022From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering30 May

Other references · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
25 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  5. 2020A Generalization of Transformer Networks to Graphscited by 653
  6. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  7. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  8. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  9. 2018Image Generation from Scene Graphscited by 324
  10. 2021Video Transformer Networkcited by 271
  11. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  12. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  13. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 165
  14. 2016Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensationcited by 156
  15. 2018ECO: Efficient Convolutional Network for Online Video Understandingcited by 115
  16. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  17. 2023Reasoning Implicit Sentiment with Chain-of-Thought Promptingcited by 79
  18. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  19. 2020What is More Likely to Happen Next? Video-and-Language Future Event Predictioncited by 61
  20. 2023Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucinationcited by 27
  21. 2023Discovering Spatio-Temporal Rationales for Video Question Answeringcited by 21
  22. 2023Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modelingcited by 16
  23. 2023Generating Visual Spatial Description via Holistic 3D Scene Understandingcited by 16
  24. 2023Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignmentcited by 15
  25. 2023Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labelingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 78; 2026: 54.

151 papers.

Sort:

By area:

  1. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  4. 2024Large Language Models Meet NLP: A Surveycited by 102
  5. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 99
  6. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  7. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  8. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  9. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  10. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  11. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  12. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
38 more
  1. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  2. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  3. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learningcited by 37
  4. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  5. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  6. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  7. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 32
  8. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  9. 2025MINERVA: Evaluating Complex Video Reasoningcited by 31
  10. 20253D Aware Region Prompted Vision Language Modelcited by 31
  11. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecturecited by 30
  12. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  13. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 29
  14. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  15. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  16. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  17. 2026Large Language Model Reasoning Failurescited by 20
  18. 2024Towards Unified Multimodal Editing with Enhanced Knowledge Collaborationcited by 18
  19. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Modelscited by 18
  20. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 17
  21. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  22. 2025Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Searchcited by 16
  23. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  24. 2024Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Networkcited by 15
  25. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 15
  26. 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15
  27. 2025Multi-Granular Multimodal Clue Fusion for Meme Understandingcited by 14
  28. 2025Taylor Series-Inspired Local Structure Fitting Network for Few-shot Point Cloud Semantic Segmentationcited by 14
  29. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  30. 2025SiLVR: A Simple Language-based Video Reasoning Frameworkcited by 14
  31. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  32. 2025Cross-modal Causal Relation Alignment for Video Question Groundingcited by 12
  33. 2025AdsQA: Towards Advertisement Video Understandingcited by 12
  34. 2024When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Codingcited by 11
  35. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  36. 2024MVTamperBench: Evaluating Robustness of Vision-Language Modelscited by 11
  37. 2025Universal Scene Graph Generationcited by 11
  38. 2025ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representationscited by 11

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning
  2. 2025Rethinking Chain-of-Thought Reasoning for Videos
  3. 2026Process-of-Thought Reasoning for Videos
  4. 2025Video Finetuning Improves Reasoning Between Frames
  5. 2025Video-QTR: Query-Driven Temporal Reasoning Framework for Lightweight Video Understanding
  6. 2023Let's Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought
  7. 2026A Very Big Video Reasoning Suite
  8. 2026TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning