mlsanityData to 7 Oct 2026

VCA: Video Curious Agent for Long Video Understanding

arXiv 2412.10471 ↗First posted 12 Dec 2024cs.CV cs.AIcited by 54

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Long video understanding poses unique challenges due to their temporal complexity and low information density. Recent works address this task by sampling numerous frames or incorporating auxiliary tools using LLMs, both of which result in high computational costs. In this work, we introduce a curiosity-driven video agent with self-exploration capability, dubbed as VCA. Built upon VLMs, VCA autonomously navigates video segments and efficiently builds a comprehensive understanding of complex video sequences. Instead of directly sampling frames, VCA employs a tree-search structure to explore video segments and collect frames. Rather than relying on external feedback or reward, VCA leverages VLM's self-generated intrinsic reward to guide its exploration, enabling it to capture the most crucial information for reasoning. Experimental results on multiple long video benchmarks demonstrate our approach's superior effectiveness and efficiency.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 66 of its 80 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

66 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning28 May
  2. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding18 Mar
  3. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agent15 Mar

Other references · 63

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
38 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  8. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  9. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  10. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  11. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  13. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  14. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  15. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  16. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  17. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  18. 2024Long Context Transfer from Language to Visioncited by 502
  19. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  20. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  21. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  22. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  23. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  24. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  25. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  26. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  27. 2023Video Understanding with Large Language Models: A Surveycited by 203
  28. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  29. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  30. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  31. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  32. 2024Pixtral 12Bcited by 174
  33. 2021Towards Long-Form Video Understandingcited by 146
  34. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  35. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  36. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  37. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  38. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 107

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 33.

54 papers.

Sort:

By area:

  1. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  2. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  3. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  4. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  5. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  6. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  7. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  8. 2025ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesiscited by 22
  9. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  10. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  11. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  12. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 14
38 more
  1. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 14
  2. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  3. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  4. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  5. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  6. 2026Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understandingcited by 8
  7. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  8. 2025AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interactioncited by 6
  9. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 6
  10. 2025AVA: Towards Agentic Video Analytics with Vision Language Modelscited by 5
  11. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  12. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 4
  13. 2025GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memorycited by 4
  14. 2025Video-QTR: Query-Driven Temporal Reasoning Framework for Lightweight Video Understandingcited by 4
  15. 2026SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaborationcited by 4
  16. 2026Where to Look Matters: On-Policy Self-Distillation for Long-Video Understandingcited by 4
  17. 2026HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answeringcited by 3
  18. 2026Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understandingcited by 3
  19. 2026Native Active Perception as Reasoning for Omni-Modal Understandingcited by 3
  20. 2025EEA: Exploration-Exploitation Agent for Long Video Understandingcited by 2
  21. 2026A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoningcited by 2
  22. 2026GLANCE: A Global-Local Coordination Multi-Agent Framework for Music-Grounded Non-Linear Video Editingcited by 2
  23. 2026LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMscited by 2
  24. 2026TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understandingcited by 2
  25. 2025SkeletonAgent: An Agentic Interaction Framework for Skeleton-based Action Recognitioncited by 1
  26. 2026Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Modelscited by 1
  27. 2026SagaQA: A Multi-hop Reasoning Benchmark for Long-form Narrative Understanding in TV Seriescited by 1
  28. 2026MAVIS: Multi-Agent Video Retrieval via Structured Video Understandingcited by 1
  29. 2026CoCoSI: Collaborative Cognitive Map Construction for Spatial Intelligencecited by 1
  30. 2026MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMscited by 1
  31. 2026AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecastingcited by 0
  32. 2026When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Modelscited by 0
  33. 2026GOPAgen: Codec-Aware Agentic Long-Video Understanding with Structured Memorycited by 0
  34. 2026From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representationscited by 0
  35. 2026QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understandingcited by 0
  36. 2026AdaVDR: Adaptive Tool Use and Reflection for Video Deep Researchcited by 0
  37. 2026VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understandingcited by 0
  38. 2026VideoResearcher: Self-Improving Tool Design for Long-Video Understandingcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Agentic Video Intelligence: A Flexible Framework for Advanced Video Exploration and Understanding
  2. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning
  3. 2026VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding
  4. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding
  5. 2026Can Agents Win the Video Browser Showdown? ↗
  6. 2026Symphony: A Cognitively-Inspired Multi-Agent System for Long-Video Understanding
  7. 2026VLX-VR: An Agentic-Aware Video Reasoning Model
  8. 2025A Challenge to Build Neuro-Symbolic Video Agents