# VideoMultiAgents: A Multi-Agent Framework for Video Question Answering

arXiv: https://arxiv.org/abs/2504.20091 · HTML: https://mlsanity.com/paper/2504.20091/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 32 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 28 of 28)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [Tarsier: Recipes for Training and Evaluating Large Video Description Models](https://mlsanity.com/paper/2407.00634/) (2024)
- [HourVideo: 1-Hour Video-Language Understanding](https://mlsanity.com/paper/2411.04998/) (2024)
- [An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM](https://mlsanity.com/paper/2403.18406/) (2024)
- [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023)
- [Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA](https://mlsanity.com/paper/2406.09396/) (2024)
- [Action Scene Graphs for Long-Form Understanding of Egocentric Videos](https://mlsanity.com/paper/2312.03391/) (2023)
- [LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos](https://mlsanity.com/paper/2312.05269/) (2023)
- [VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMs](https://mlsanity.com/paper/2409.20365/) (2024)
- [LinVT: Empower Your Image-level Large Language Model to Understand Videos](https://mlsanity.com/paper/2412.05185/) (2024)
- [Learning Situation Hyper-Graphs for Video Question Answering](https://mlsanity.com/paper/2304.08682/) (2023)
- [TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models](https://mlsanity.com/paper/2411.11066/) (2024)
- [ENTER: Event Based Interpretable Reasoning for VideoQA](https://mlsanity.com/paper/2501.14194/) (2025)
- [VDMA: Video Question Answering with Dynamically Generated Multi-Agents](https://mlsanity.com/paper/2407.03610/) (2024)
- [Towards Fine-Grained Video Question Answering](https://mlsanity.com/paper/2503.06820/) (2025)

## Built on by (most cited first, 16 of 16)

- [StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding](https://mlsanity.com/paper/2508.01875/) (2025)
- [VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoning](https://mlsanity.com/paper/2506.06097/) (2025)
- [VideoExplorer: Think With Videos For Agentic Long-Video Understanding](https://mlsanity.com/paper/2506.10821/) (2025)
- [Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding](https://mlsanity.com/paper/2512.05774/) (2025)
- [UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist](https://mlsanity.com/paper/2511.08521/) (2025)
- [MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation](https://mlsanity.com/paper/2506.17113/) (2025)
- [VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning](https://mlsanity.com/paper/2603.25021/) (2026)
- [EEA: Exploration-Exploitation Agent for Long Video Understanding](https://mlsanity.com/paper/2512.03500/) (2025)
- [DIVE: Deep-search Iterative Video Exploration A Technical Report for the CVRR Challenge at CVPR 2025](https://mlsanity.com/paper/2506.21891/) (2025)
- [Structured Prompting and Multi-Agent Knowledge Distillation for Traffic Video Interpretation and Risk Inference](https://mlsanity.com/paper/2508.13439/) (2025)
- [AVATAAR: Agentic Video Answering via Temporal Adaptive Alignment and Reasoning](https://mlsanity.com/paper/2511.15578/) (2025)
- [Diagnosing Failure Modes of Shared-State Collaboration in Resource-Constrained Visual Agents](https://mlsanity.com/paper/2605.31354/) (2026)
- [Child-Oriented AIGC Video Risk Reviewing: A Benchmark and Knowledge-Supported Iterative Reasoning Framework](https://mlsanity.com/paper/2607.22715/) (2026)
- [TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding](https://mlsanity.com/paper/2608.25935/) (2026)
- [CAM: Question Answering on Entity-Centric Videos with Continuous Extraction and Adaptive Querying](https://mlsanity.com/paper/2609.06504/) (2026)
- [AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents](https://mlsanity.com/paper/2609.21386/) (2026)

## Abstract (arXiv)

Video Question Answering (VQA) inherently relies on multimodal reasoning, integrating visual, temporal, and linguistic cues to achieve a deeper understanding of video content. However, many existing methods rely on feeding frame-level captions into a single model, making it difficult to adequately capture temporal and interactive contexts. To address this limitation, we introduce VideoMultiAgents, a framework that integrates specialized agents for vision, scene graph analysis, and text processing. It enhances video understanding leveraging complementary multimodal reasoning from independently operating agents. Our approach is also supplemented with a question-guided caption generation, which produces captions that highlight objects, actions, and temporal transitions directly relevant to a given query, thus improving the answer accuracy. Experimental results demonstrate that our method achieves state-of-the-art performance on Intent-QA (79.0%, +6.2% over previous SOTA), EgoSchema subset (75.4%, +3.4%), and NExT-QA (79.6%, +0.4%). The source code is available at https://github.com/PanasonicConnect/VideoMultiAgents.