mlsanityData to 7 Oct 2026

VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding

arXiv 2403.11481 ↗First posted 18 Mar 2024cs.CVcited by 224

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

also in 1 more
Abstract (arXiv)

We explore how reconciling several foundation models (large language models and vision-language models) with a novel unified memory mechanism could tackle the challenging video understanding problem, especially capturing the long-term temporal relations in lengthy videos. In particular, the proposed multimodal agent VideoAgent: 1) constructs a structured memory to store both the generic temporal event descriptions and object-centric tracking states of the video; 2) given an input task query, it employs tools including video segment localization and object memory querying along with other visual foundation models to interactively solve the task, utilizing the zero-shot tool-use ability of LLMs. VideoAgent demonstrates impressive performances on several long-horizon video understanding benchmarks, an average increase of 6.6% on NExT-QA and 26.0% on EgoSchema over baselines, closing the gap between open-sourced models and private counterparts including Gemini 1.5 Pro.

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 50 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

47 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridge25 Feb
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
32 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  4. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  8. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 526
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  11. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  12. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  13. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
  14. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 234
  15. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  16. 2022Learning Video Representations from Large Language Modelscited by 204
  17. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  18. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  19. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  20. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  21. 2021Towards Long-Form Video Understandingcited by 146
  22. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  23. 2023MindAgent: Emergent Gaming Interactioncited by 87
  24. 2022Temporal Alignment Networks for Long-term Videocited by 84
  25. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  26. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  27. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 46
  28. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  29. 2023EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understandingcited by 39
  30. 2023Text-Conditioned Resampler For Long Form Video Understandingcited by 24
  31. 2023GroundNLQ @ Ego4D Natural Language Queries Challenge 2023cited by 20
  32. 2022Compressed Vision for Efficient Video Understandingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 99; 2026: 94.

224 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  3. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  4. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  5. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  6. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  7. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  8. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  9. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  10. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  11. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  12. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
38 more
  1. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  2. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  3. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  4. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  5. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  6. 2024Language Repository for Long Video Understandingcited by 55
  7. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  8. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  9. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  10. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  11. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  12. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  13. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  14. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  15. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  16. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  17. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  18. 2024OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquercited by 34
  19. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  20. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 31
  21. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  22. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 30
  23. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  24. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  25. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  26. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  27. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  28. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  29. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  30. 2025LongVideoAgent: Multi-Agent Reasoning with Long Videoscited by 23
  31. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  32. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  33. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  34. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  35. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  36. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19
  37. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  38. 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Modelscited by 18

50 most cited of 224 All 224 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agent
  2. 2026VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding
  3. 2026Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory
  4. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use
  5. 2025Agentic Video Intelligence: A Flexible Framework for Advanced Video Exploration and Understanding
  6. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline
  7. 2025Infinite Video Understanding
  8. 2026VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning