mlsanityData to 14 Oct 2026

VideoRAG: Retrieval-Augmented Generation over Video Corpus

arXiv 2501.05874 ↗First posted 10 Jan 2025cs.CV cs.AI cs.CLcited by 57

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) is a powerful strategy for improving the factual accuracy of models by retrieving external knowledge relevant to queries and incorporating it into the generation process. However, existing approaches primarily focus on text, with some recent advancements considering images, and they largely overlook videos, a rich source of multimodal knowledge capable of representing contextual details more effectively than any other modality. While very recent studies explore the use of videos in response generation, they either predefine query-associated videos without retrieval or convert videos into textual descriptions losing multimodal richness. To tackle these, we introduce VideoRAG, a framework that not only dynamically retrieves videos based on their relevance with queries but also utilizes both visual and textual information. The operation of VideoRAG is powered by recent Large Video Language Models (LVLMs), which enable the direct processing of video content to represent it for retrieval and the seamless integration of retrieved videos jointly with queries for response generation. Also, inspired by that the context size of LVLMs may not be sufficient to process all frames in extremely long videos and not all frames are equally important, we introduce a video frame selection mechanism to extract the most informative subset of frames, along with a strategy to extract textual information from videos (as it can aid the understanding of video content) when their subtitles are not available. We experimentally validate the effectiveness of VideoRAG, showcasing that it is superior to relevant baselines. Code is available at https://github.com/starsuzi/VideoRAG.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 45 of its 68 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

45 of 68 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension20 Nov
  2. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications29 Oct
  3. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents14 Oct
  4. 2024iRAG: Advancing RAG for Videos with an Incremental Approach18 Apr
  5. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov
  6. 2022Retrieval Augmented Visual Question Answering with Outside Knowledge7 Oct
  7. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
26 more
  1. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  2. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  3. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  4. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  6. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  7. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  8. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  9. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  10. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  11. 2023Video Understanding with Large Language Models: A Surveycited by 206
  12. 2024An Introduction to Vision-Language Modelingcited by 178
  13. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  14. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  15. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 93
  16. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  17. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 65
  18. 2024Elysium: Exploring Object-level Perception in Videos via MLLMcited by 48
  19. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  20. 2023Retrieval-Augmented Code Generation for Universal Information Extractioncited by 40
  21. 2024Text2SQL is Not Enough: Unifying AI and Databases with TAGcited by 34
  22. 2023Retrieval-Augmented Text-to-Audio Generationcited by 24
  23. 2024OmniVid: A Generative Framework for Universal Video Understandingcited by 22
  24. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  25. 2024Database-Augmented Query Representation for Information Retrievalcited by 5
  26. 2024Towards Effective Counter-Responses: Aligning Human Preferences with Strategies to Combat Online Trollingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 30.

57 papers.

Sort:

By area:

  1. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  2. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 53
  3. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  4. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  5. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 38
  6. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  7. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  8. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 20
  9. 2025ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understandingcited by 16
  10. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 12
  11. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  12. 2026Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Searchcited by 8
38 more
  1. 2025RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graphcited by 7
  2. 2025SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understandingcited by 7
  3. 2026MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanismcited by 7
  4. 2025Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Modelcited by 6
  5. 2025MARC: Memory-Augmented RL Token Compression for Efficient Video Understandingcited by 6
  6. 2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
  7. 2026VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAGcited by 6
  8. 2026VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graphcited by 5
  9. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agentscited by 5
  10. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 5
  11. 2025GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memorycited by 4
  12. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  13. 2025AMAQA: A Metadata-based QA Dataset for RAG Systemscited by 2
  14. 2025VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generationcited by 2
  15. 2025X-CoT: Explainable Text-to-Video Retrieval via LLM-based Chain-of-Thought Reasoningcited by 2
  16. 2025MCA: Modality Composition Awareness for Robust Composed Multimodal Retrievalcited by 2
  17. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  18. 2026VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understandingcited by 2
  19. 2026Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understandingcited by 2
  20. 2025RAVID: Retrieval-Augmented Visual Detection: A Knowledge-Driven Approach for AI-Generated Image Identificationcited by 1
  21. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  22. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
  23. 2026RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systemscited by 1
  24. 2026ForeSea: AI Forensic Search with Multi-modal Queries for Video Surveillancecited by 1
  25. 2026CRAFT: Critic-Refined Adaptive Key-Frame Targeting for Multimodal Video Question Answeringcited by 1
  26. 2026Driving Video Retrieval for Complex Queries with Structured Groundingcited by 1
  27. 2026Rethinking RAG in Long Videos: What to Retrieve and How to Use It?cited by 1
  28. 2026EM^2Mem: Event-Centric Multimodal Memory for Large Language Modelscited by 1
  29. 2025Atom: Efficient On-Device Video-Language Pipelines Through Modular Reusecited by 0
  30. 2025M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generationcited by 0
  31. 2026Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmarkcited by 0
  32. 2026UpstreamQA: A Modular Framework for Explicit Reasoning on Video Question Answering Taskscited by 0
  33. 2026OmniRetrieval: Unified Retrieval across Heterogeneous Knowledge Sourcescited by 0
  34. 2026GOPAgen: Codec-Aware Agentic Long-Video Understanding with Structured Memorycited by 0
  35. 2026HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoningcited by 0
  36. 2026CHaystack: Benchmarking Chinese Document Retrieval and VQAcited by 0
  37. 2026Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidencecited by 0
  38. 2026Coarse Indexing, Fine Evidence: Decoupling Temporal Granularity in Long-Video RAGcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos
  2. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding
  3. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  4. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries
  5. 2024Video Enriched Retrieval Augmented Generation Using Aligned Video Captions
  6. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generation
  7. 2025RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understanding
  8. 2025E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generation