# LongVideoAgent: Multi-Agent Reasoning with Long Videos

arXiv: https://arxiv.org/abs/2512.20618 · HTML: https://mlsanity.com/paper/2512.20618/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling](https://mlsanity.com/paper/2501.12386/) (2025)
- [Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension](https://mlsanity.com/paper/2411.13093/) (2024)
- [SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models](https://mlsanity.com/paper/2407.15841/) (2024)
- [MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens](https://mlsanity.com/paper/2404.03413/) (2024)
- [Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization](https://mlsanity.com/paper/2402.03161/) (2024)
- [Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory](https://mlsanity.com/paper/2508.09736/) (2025)
- [Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering](https://mlsanity.com/paper/2402.10698/) (2024)

## Built on by (most cited first, 25 of 25)

- [ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning](https://mlsanity.com/paper/2605.20342/) (2026)
- [HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering](https://mlsanity.com/paper/2603.18558/) (2026)
- [Scaling Video Understanding via Compact Latent Multi-Agent Collaboration](https://mlsanity.com/paper/2605.00444/) (2026)
- [Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning](https://mlsanity.com/paper/2605.08271/) (2026)
- [Divide-then-Diagnose: Weaving Clinician-Inspired Contexts for Ultra-Long Capsule Endoscopy Videos](https://mlsanity.com/paper/2604.21814/) (2026)
- [Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning](https://mlsanity.com/paper/2605.28192/) (2026)
- [MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling](https://mlsanity.com/paper/2602.13332/) (2026)
- [GLANCE: A Global-Local Coordination Multi-Agent Framework for Music-Grounded Non-Linear Video Editing](https://mlsanity.com/paper/2604.05076/) (2026)
- [See More, Think Deeper: Query-Expanded Visual Evidence and Answer-Clue Guided Reflection for Long Video Understanding](https://mlsanity.com/paper/2606.09064/) (2026)
- [EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent](https://mlsanity.com/paper/2607.13792/) (2026)
- [AgenticVAU: Multi-Agent Explore-Verify Reasoning for Video Anomaly Understanding](https://mlsanity.com/paper/2608.03779/) (2026)
- [DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection](https://mlsanity.com/paper/2605.23281/) (2026)
- [Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?](https://mlsanity.com/paper/2606.08063/) (2026)
- [CoCoSI: Collaborative Cognitive Map Construction for Spatial Intelligence](https://mlsanity.com/paper/2606.10401/) (2026)
- [Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory](https://mlsanity.com/paper/2607.05511/) (2026)
- [Coverage-Driven Adaptive Keyframe Selection for Video Understanding](https://mlsanity.com/paper/2608.00714/) (2026)
- [Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviors](https://mlsanity.com/paper/2604.03631/) (2026)
- [Accuracy and Cost Claims Do Not Survive Re-Execution in Agentic VideoQA](https://mlsanity.com/paper/2607.01469/) (2026)
- [Statistical Adversaries: Natural Backdoor-like Adversarial Features in Clean Vision Datasets](https://mlsanity.com/paper/2607.05516/) (2026)
- [Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents](https://mlsanity.com/paper/2607.11433/) (2026)
- [CADER: Confidence-Aware Dynamic Evidence Reasoning for Long-Video Understanding](https://mlsanity.com/paper/2607.24582/) (2026)
- [From Intent to Evidence: Policy-Steered Multi-Strategy Retrieval for Long-Video Agents](https://mlsanity.com/paper/2608.31005/) (2026)
- [Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models](https://mlsanity.com/paper/2609.28991/) (2026)
- [VideoLoop: Looped Working Memory Against Semantic Thrashing in Long-Form Video Agents](https://mlsanity.com/paper/2609.38119/) (2026)
- [CASE: Cost-Aware Stopping for Efficient Long-Video Agents](https://mlsanity.com/paper/2610.05400/) (2026)

## Abstract (arXiv)

Recent advances in multimodal LLMs and systems that use tools for long-video QA point to the promise of reasoning over hour-long episodes. However, many methods still compress content into lossy summaries or rely on limited toolsets, weakening temporal grounding and missing fine-grained cues. We propose a multi-agent framework in which a master LLM coordinates a grounding agent to localize question-relevant segments and a vision agent to extract targeted textual observations. The master agent plans with a step limit, and is trained with reinforcement learning to encourage concise, correct, and efficient multi-agent cooperation. This design helps the master agent focus on relevant clips via grounding, complements subtitles with visual detail, and yields interpretable trajectories. On our proposed LongTVQA and LongTVQA+ which are episode-level datasets aggregated from TVQA/TVQA+, our multi-agent system significantly outperforms strong non-agent baselines. Experiments also show reinforcement learning further strengthens reasoning and planning for the trained agent. Code and data will be shared at https://longvideoagent.github.io/.