mlsanityData to 7 Oct 2026

VideoAgent: Long-form Video Understanding with Large Language Model as Agent

arXiv 2403.10517 ↗First posted 15 Mar 2024cs.CV cs.AI cs.CLcited by 323

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding.

Key facts. Cited by 323 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 57 of its 69 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

57 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)16 Jan

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2024Mixtral of Expertscited by 1,785
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  6. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  11. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  12. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  13. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  14. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  15. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  16. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  17. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  18. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  19. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  20. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  21. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  22. 2022Learning Video Representations from Large Language Modelscited by 204
  23. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  24. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  25. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  26. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  27. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  28. 2015End-to-end Learning of Action Detection from Frame Glimpses in Videoscited by 101
  29. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  30. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  31. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  32. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  33. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  34. 2019VideoGraph: Recognizing Minutes-Long Human Activities in Videoscited by 57
  35. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  36. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  37. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  38. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 142; 2026: 140.

323 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  3. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  4. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  5. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  6. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  7. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  8. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  9. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  10. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  11. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  12. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
38 more
  1. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  2. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  3. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  4. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  5. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 80
  6. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  7. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  8. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  9. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  10. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  11. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  12. 2024Language Repository for Long Video Understandingcited by 55
  13. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  14. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  15. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  16. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  17. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 43
  18. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  19. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 40
  20. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  21. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  22. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  23. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  24. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  25. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  26. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 30
  27. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  28. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  29. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  30. 2025VideoExplorer: Think With Videos For Agentic Long-Video Understandingcited by 29
  31. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  32. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  33. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  34. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  35. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  36. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  37. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  38. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25

50 most cited of 323 All 323 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
  2. 2024OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer
  3. 2025Agentic Video Intelligence: A Flexible Framework for Advanced Video Exploration and Understanding
  4. 2026VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning
  5. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
  6. 2026VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding
  7. 2025CAViAR: Critic-Augmented Video Agentic Reasoning
  8. 2026VideoAgent: All-in-One Framework for Video Understanding and Editing