mlsanityData to 7 Oct 2026

Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation

arXiv 2412.01694 ↗First posted 2 Dec 2024cs.CVcited by 36

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

This paper tackles the problem of video question answering (VideoQA), a task that often requires multi-step reasoning and a profound understanding of spatial-temporal dynamics. While large video-language models perform well on benchmarks, they often lack explainability and spatial-temporal grounding. In this paper, we propose Agent-of-Thoughts Distillation (AoTD), a method that enhances models by incorporating automatically generated Chain-of-Thoughts (CoTs) into the instruction-tuning process. Specifically, we leverage an agent-based system to decompose complex questions into sub-tasks, and address them with specialized vision models, the intermediate results are then treated as reasoning chains. We also introduce a verification mechanism using a large language model (LLM) to ensure the reliability of generated CoTs. Extensive experiments demonstrate that AoTD improves the performance on multiple-choice and open-ended benchmarks.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 60 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

50 of 60 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition7 May
  2. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLM24 Apr
  3. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answering9 Apr
  4. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  5. 2024VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding21 Mar
  6. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models5 Dec
  7. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  8. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Models23 May
  9. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  10. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  11. 2022Visual Programming: Compositional visual reasoning without training18 Nov

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
27 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  4. 2021Grounded Language-Image Pre-trainingcited by 1,147
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  9. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  10. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  12. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  13. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  14. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  15. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  16. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  17. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  18. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  19. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  20. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  21. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  22. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  23. 2023Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimizationcited by 54
  24. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  25. 2024Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervisioncited by 24
  26. 2023ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videoscited by 15
  27. 2024Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationalescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 12.

36 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  3. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 32
  4. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  5. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  6. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  7. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  8. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  9. 2025ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesiscited by 22
  10. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  11. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 15
  12. 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15
24 more
  1. 2025Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesiscited by 8
  2. 2025CountGD++: Generalized Prompting for Open-World Countingcited by 8
  3. 2026Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understandingcited by 8
  4. 2025Debunk and Infer: Multimodal Fake News Detection via Diffusion-Generated Evidence and LLM Reasoningcited by 7
  5. 2025Merge-of-Thought Distillationcited by 6
  6. 2025FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoningcited by 5
  7. 2025Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graphcited by 5
  8. 2025Video-Skill-CoT: Skill-based Chain-of-Thoughts for Domain-Adaptive Video Reasoningcited by 3
  9. 2025CoTasks: Chain-of-Thought based Video Instruction Tuning Taskscited by 3
  10. 2025NeMo: Needle in a Montage for Video-Language Understandingcited by 3
  11. 2026CurEvo: Curriculum-Guided Self-Evolution for Video Understandingcited by 3
  12. 2025CompAgent: An Agentic Framework for Visual Compliance Verificationcited by 1
  13. 2025DuoTeach: Dual Role Self-Teaching for Coarse-to-Fine Decision Coordination in Vision--Language Modelscited by 1
  14. 2026Weaver: End-to-End Agentic System Training for Video Interleaved Reasoningcited by 1
  15. 2026Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoningcited by 1
  16. 2025Lightweight Structured Multimodal Reasoning for Clinical Scene Understanding in Roboticscited by 0
  17. 2026Enhancing Lightweight Vision Language Models through Group Competitive Learning for Socially Compliant Navigationcited by 0
  18. 2026AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecastingcited by 0
  19. 2026CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answeringcited by 0
  20. 2026From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representationscited by 0
  21. 2026EFlow: Learning Evidence Flow for Long-Video Reasoning with Adaptive Reflectioncited by 0
  22. 2026Long-to-Short Video Evidence Reasoning for Grounded Question Answeringcited by 0
  23. 2026Test-time Reinforcement Learning for Anomalous Video Understandingcited by 0
  24. 2026Agentic Video Understanding: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Reasoning Distillation: Small Video-Language Models via Synthetic CoT and Difficulty-Aware Fine-Tuning
  2. 2026VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twice
  3. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning
  4. 2025Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning
  5. 2026Sprout: Building Dynamic Memory While Reasoning for Agentic Video Understanding
  6. 2025Rethinking Chain-of-Thought Reasoning for Videos
  7. 2026AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents
  8. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning