mlsanityData to 7 Oct 2026

VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection

arXiv 2411.14794 ↗First posted 22 Nov 2024cs.CV cs.AI cs.CLcited by 64

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The advancement of Large Vision Language Models (LVLMs) has significantly improved multimodal understanding, yet challenges remain in video reasoning tasks due to the scarcity of high-quality, large-scale datasets. Existing video question-answering (VideoQA) datasets often rely on costly manual annotations with insufficient granularity or automatic construction methods with redundant frame-by-frame analysis, limiting their scalability and effectiveness for complex reasoning. To address these challenges, we introduce VideoEspresso, a novel dataset that features VideoQA pairs preserving essential spatial details and temporal coherence, along with multimodal annotations of intermediate reasoning steps. Our construction pipeline employs a semantic-aware method to reduce redundancy, followed by generating QA pairs using GPT-4o. We further develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes, guiding GPT-4o in extracting logical relationships from QA pairs and video content. To exploit the potential of high-quality VideoQA pairs, we propose a Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a two-stage instruction fine-tuned reasoning LVLM. This framework adaptively selects core frames and performs CoT reasoning using multimodal evidence. Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our method outperforms existing baselines on most tasks, demonstrating superior video reasoning capabilities. Our code and dataset will be released at: https://github.com/hshjerry/VideoEspresso

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 56 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

44 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool7 Jul
  2. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition7 May
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr
  4. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
27 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  6. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  8. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  9. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  11. 2024Long Context Transfer from Language to Visioncited by 502
  12. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  13. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  14. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  15. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  16. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  17. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  18. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  19. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  20. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  21. 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 235
  22. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  23. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  24. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  25. 2024Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodologycited by 92
  26. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  27. 2024Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomalycited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 36; 2026: 27.

64 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  4. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  5. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  6. 2025Scaling RL to Long Videoscited by 83
  7. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  8. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  9. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  10. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  11. 2025Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoningcited by 36
  12. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
38 more
  1. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  2. 2025MINERVA: Evaluating Complex Video Reasoningcited by 31
  3. 2025RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluationcited by 31
  4. 2025ChestX-Reasoner: Advancing Radiology Foundation Models with Reasoning through Step-by-Step Verificationcited by 30
  5. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  6. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  7. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  8. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 25
  9. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  10. 2025FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learningcited by 20
  11. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 19
  12. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processingcited by 18
  13. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Modelscited by 18
  14. 2025Factuality Matters: When Image Generation and Editing Meet Structured Visualscited by 16
  15. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 15
  16. 2025PICABench: How Far Are We from Physically Realistic Image Editing?cited by 13
  17. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  18. 2025DAVID-XR1: Detecting AI-Generated Videos with Explainable Reasoningcited by 8
  19. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  20. 2026LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understandingcited by 8
  21. 2026Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisitioncited by 8
  22. 2025StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQAcited by 6
  23. 2025KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understandingcited by 6
  24. 2026Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Modelscited by 6
  25. 2026SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenancecited by 6
  26. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 6
  27. 2025MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoningcited by 5
  28. 2026VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understandingcited by 5
  29. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  30. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5
  31. 2025A Unified Multi-Agent Framework for Universal Multimodal Understanding and Generationcited by 4
  32. 2026Cut to the Chase: Training-free Multimodal Summarization via Chain-of-Eventscited by 3
  33. 2026VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Promptingcited by 3
  34. 2026Video-Oasis: Rethinking Evaluation of Video Understandingcited by 3
  35. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  36. 2026Sekai2: From World Exploration to Interactive World Modelingcited by 2
  37. 2025VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMscited by 1
  38. 2026Weaver: End-to-End Agentic System Training for Video Interleaved Reasoningcited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026UpstreamQA: A Modular Framework for Explicit Reasoning on Video Question Answering Tasks
  2. 2025Video Finetuning Improves Reasoning Between Frames
  3. 2025Video-QTR: Query-Driven Temporal Reasoning Framework for Lightweight Video Understanding
  4. 2025ReasVQA: Advancing VideoQA with Imperfect Reasoning Process
  5. 2024Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
  6. 2023ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videos
  7. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  8. 2025Rethinking Chain-of-Thought Reasoning for Videos