mlsanityData to 14 Oct 2026

DrVideo: Document Retrieval Based Long Video Understanding

arXiv 2406.12846 ↗First posted 18 Jun 2024cs.CVcited by 65

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most of the existing methods for video understanding primarily focus on videos only lasting tens of seconds, with limited exploration of techniques for handling long videos. The increased number of frames in long videos poses two main challenges: difficulty in locating key information and performing long-range reasoning. Thus, we propose DrVideo, a document-retrieval-based system designed for long video understanding. Our key idea is to convert the long-video understanding problem into a long-document understanding task so as to effectively leverage the power of large language models. Specifically, DrVideo first transforms a long video into a coarse text-based long document to initially retrieve key frames and then updates the documents with the augmented key frame information. It then employs an agent-based iterative loop to continuously search for missing information and augment the document until sufficient question-related information is gathered for making the final predictions in a chain-of-thought manner. Extensive experiments on long video benchmarks confirm the effectiveness of our method. DrVideo significantly outperforms existing LLM-based state-of-the-art methods on EgoSchema benchmark (3 minutes), MovieChat-1K benchmark (10 minutes), and the long split of Video-MME benchmark (average of 44 minutes).

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 64 of its 85 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

64 of 85 references.

References · 64

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023Mistral 7Bcited by 3,164
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  11. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  12. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  13. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  14. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  15. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  16. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  17. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  18. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  19. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  20. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  21. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  22. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  23. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  24. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  25. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  26. 2022Learning Video Representations from Large Language Modelscited by 205
  27. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  28. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  29. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  30. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  31. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  32. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  33. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  34. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  35. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  36. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  37. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  38. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 28; 2026: 33.

65 papers.

Sort:

By area:

  1. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  2. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 77
  3. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 57
  4. 2024VCA: Video Curious Agent for Long Video Understandingcited by 56
  5. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  6. 2024AMEGO: Active Memory from long EGOcentric videoscited by 36
  7. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  8. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 31
  9. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 23
  10. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 22
  11. 2025UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarkscited by 19
  12. 2025Memory-enhanced Retrieval Augmentation for Long Video Understandingcited by 17
38 more
  1. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 17
  2. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 15
  3. 2025SiLVR: A Simple Language-based Video Reasoning Frameworkcited by 14
  4. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  5. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Usecited by 12
  6. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  7. 2026Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Searchcited by 8
  8. 2026Event-Anchored Frame Selection for Effective Long-Video Understandingcited by 7
  9. 2024ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAGcited by 6
  10. 2025TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understandingcited by 6
  11. 2026VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAGcited by 6
  12. 2026VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authoritycited by 6
  13. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 5
  14. 2025AVA: Towards Agentic Video Analytics with Vision Language Modelscited by 5
  15. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Samplingcited by 5
  16. 2025An Empirical Study on How Video-LLMs Answer Video Questionscited by 4
  17. 2025Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learningcited by 4
  18. 2025GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memorycited by 4
  19. 2025Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understandingcited by 4
  20. 2025Video-QTR: Query-Driven Temporal Reasoning Framework for Lightweight Video Understandingcited by 4
  21. 2026Native Active Perception as Reasoning for Omni-Modal Understandingcited by 4
  22. 2025A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answeringcited by 3
  23. 2026VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoningcited by 3
  24. 2026Video-Oasis: Rethinking Evaluation of Video Understandingcited by 3
  25. 2026Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrievalcited by 3
  26. 2025CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generationcited by 2
  27. 2025EEA: Exploration-Exploitation Agent for Long Video Understandingcited by 2
  28. 2025Towards Effective Long Video Understanding of Multimodal Large Language Models via One-shot Clip Retrievalcited by 2
  29. 2026A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoningcited by 2
  30. 2026Progressive Video Condensation with MLLM Agent for Long-form Video Understandingcited by 2
  31. 2026Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understandingcited by 2
  32. 2026Online Video Agent Harness for Long Video Understandingcited by 2
  33. 2025See What You Need: Query-Aware Visual Intelligence through Reasoning-Perception Loopscited by 1
  34. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  35. 2026Adaptive Greedy Frame Selection for Long Video Understandingcited by 1
  36. 2026ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Groundingcited by 1
  37. 2026Rethinking RAG in Long Videos: What to Retrieve and How to Use It?cited by 1
  38. 2025VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answeringcited by 0

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Text-to-Video Retrieval via Digital Twin Video Representations and Large Language Models
  2. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  3. 2026RANKVIDEO: Reasoning Reranking for Text-to-Video Retrieval
  4. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding
  5. 2025E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generation
  6. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  7. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding
  8. 2024LongVLM: Efficient Long Video Understanding via Large Language Models