mlsanityData to 7 Oct 2026

Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding

arXiv 2505.18079 ↗First posted 23 May 2025cs.CV cs.AI cs.CLcited by 94

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Long-form video understanding presents significant challenges due to extensive temporal-spatial complexity and the difficulty of question answering under such extended contexts. While Large Language Models (LLMs) have demonstrated considerable advancements in video analysis capabilities and long context handling, they continue to exhibit limitations when processing information-dense hour-long videos. To overcome such limitations, we propose the Deep Video Discovery (DVD) agent to leverage an agentic search strategy over segmented video clips. Unlike previous video agents that rely on predefined workflows applied uniformly across different queries, our approach emphasizes the autonomous and adaptive nature of agents. By providing a set of search-centric tools on multi-granular video database, our DVD agent leverages the advanced reasoning capability of LLM to plan on its current observation state, strategically selects tools to orchestrate adaptive workflow for different queries in light of the gathered information. We perform comprehensive evaluation on multiple long video understanding benchmarks that demonstrates our advantage. Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of 74.2%, which substantially surpasses all prior works, and further improves to 76.0% with transcripts. The code has been released at https://github.com/microsoft/DeepVideoDiscovery.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 38 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

29 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024VCA: Video Curious Agent for Long Video Understanding12 Dec
  2. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding18 Mar
  3. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agent15 Mar

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  9. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  10. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  11. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  12. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
14 more
  1. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  2. 2023Tool Learning with Foundation Modelscited by 302
  3. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  4. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  5. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  6. 2024Tool Learning with Large Language Models: A Surveycited by 221
  7. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  8. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  9. 2025Qwen2.5-1M Technical Reportcited by 158
  10. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  11. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  12. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  13. 2023Responsible Task Automation: Empowering Large Language Models as Responsible Task Automatorscited by 14
  14. 2024DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understandingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 80.

94 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 39
  3. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  4. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  5. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  6. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 24
  7. 2025ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesiscited by 22
  8. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  9. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  10. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
  11. 2025TeleMem: Building Long-Term and Multimodal Memory for Agentic AIcited by 14
  12. 2026Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoningcited by 14
38 more
  1. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 14
  2. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 14
  3. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  4. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  5. 2026Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Searchcited by 8
  6. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  7. 2026Symphony: A Cognitively-Inspired Multi-Agent System for Long-Video Understandingcited by 8
  8. 2026VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoningcited by 7
  9. 2025Agentic Video Intelligence: A Flexible Framework for Advanced Video Exploration and Understandingcited by 6
  10. 2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
  11. 2026VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verificationcited by 6
  12. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  13. 2026VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authoritycited by 5
  14. 2026ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learningcited by 5
  15. 2026VideoAgent: All-in-One Framework for Video Understanding and Editingcited by 5
  16. 2026SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaborationcited by 4
  17. 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaborationcited by 4
  18. 2026Where to Look Matters: On-Policy Self-Distillation for Long-Video Understandingcited by 4
  19. 2026Video-Oasis: Rethinking Evaluation of Video Understandingcited by 3
  20. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 3
  21. 2026Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoningcited by 3
  22. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  23. 2026Native Active Perception as Reasoning for Omni-Modal Understandingcited by 3
  24. 2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
  25. 2026VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Modelscited by 3
  26. 2025Infinite Video Understandingcited by 2
  27. 2025EEA: Exploration-Exploitation Agent for Long Video Understandingcited by 2
  28. 2026A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoningcited by 2
  29. 2026Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoningcited by 2
  30. 2026GLANCE: A Global-Local Coordination Multi-Agent Framework for Music-Grounded Non-Linear Video Editingcited by 2
  31. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solverscited by 2
  32. 2026MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understandingcited by 2
  33. 2026TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understandingcited by 2
  34. 2026Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understandingcited by 2
  35. 2025ChronoForge-RL: Chronological Forging through Reinforcement Learning for Enhanced Video Understandingcited by 1
  36. 2026AD-MIR: Bridging the Gap from Perception to Persuasion in Advertising Video Understanding via Structured Reasoningcited by 1
  37. 2026VideoAtlas: Navigating Long-Form Video in Logarithmic Computecited by 1
  38. 2026VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understandingcited by 1

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning
  2. 2026Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understanding
  3. 2026RACER: Reflective Agent Coupling Query Interpretation and Tool-Based Retrieval for Frame Selection in Long Video Understanding
  4. 2026VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding
  5. 2026VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning
  6. 2025AVATAAR: Agentic Video Answering via Temporal Adaptive Alignment and Reasoning
  7. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding
  8. 2026Incentivizing Vision Language Models to Search for Long Video Question Answering