# iRAG: Advancing RAG for Videos with an Incremental Approach arXiv: https://arxiv.org/abs/2404.12309 ยท HTML: https://mlsanity.com/paper/2404.12309/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. - Builds on 13 of its 32 references that are in the map. - First posted on arXiv 18 Apr 2024. ## Placed in - Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/) ## Builds on (most cited first, 13 of 13) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020) - [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017) - [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018) - [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021) - [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021) - [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023) - [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021) - [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023) - [LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling](https://mlsanity.com/paper/2206.07160/) (2022) - [GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features](https://mlsanity.com/paper/2207.09666/) (2022) - [LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMs](https://mlsanity.com/paper/2309.00841/) (2023) - [Extract-Transform-Load for Video Streams](https://mlsanity.com/paper/2310.04830/) (2023) ## Built on by (most cited first, 15 of 15) - [VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos](https://mlsanity.com/paper/2502.01549/) (2025) - [VideoRAG: Retrieval-Augmented Generation over Video Corpus](https://mlsanity.com/paper/2501.05874/) (2025) - [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025) - [OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer](https://mlsanity.com/paper/2406.16620/) (2024) - [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025) - [ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAG](https://mlsanity.com/paper/2411.07688/) (2024) - [Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.00028/) (2025) - [Towards Retrieval Augmented Generation over Large Video Libraries](https://mlsanity.com/paper/2406.14938/) (2024) - [On the Effectiveness of Proposed Techniques to Reduce Energy Consumption in RAG Systems: A Controlled Experiment](https://mlsanity.com/paper/2601.02522/) (2026) - [VRAG-DFD: Verifiable Retrieval-Augmentation for MLLM-based Deepfake Detection](https://mlsanity.com/paper/2604.13660/) (2026) - [Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding](https://mlsanity.com/paper/2608.07663/) (2026) - [CMRAG: Co-modality-based visual document retrieval and question answering](https://mlsanity.com/paper/2509.02123/) (2025) - [ELLMPEG: An Edge-based Agentic LLM Video Processing Tool](https://mlsanity.com/paper/2602.00028/) (2026) - [Open-SAT: LLM-Guided Query Embedding Refinement for Open-Vocabulary Object Retrieval in Satellite Imagery](https://mlsanity.com/paper/2605.05344/) (2026) - [CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2610.03421/) (2026) ## Abstract (arXiv) Retrieval-augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for understanding of videos is appealing but there are two critical limitations. One-time, upfront conversion of all content in large corpus of videos into text descriptions entails high processing times. Also, not all information in the rich video data is typically captured in the text descriptions. Since user queries are not known apriori, developing a system for video to text conversion and interactive querying of video data is challenging. To address these limitations, we propose an incremental RAG system called iRAG, which augments RAG with a novel incremental workflow to enable interactive querying of a large corpus of videos. Unlike traditional RAG, iRAG quickly indexes large repositories of videos, and in the incremental workflow, it uses the index to opportunistically extract more details from select portions of the videos to retrieve context relevant to an interactive user query. Such an incremental workflow avoids long video to text conversion times, and overcomes information loss issues due to conversion of video to text, by doing on-demand query-specific extraction of details in video data. This ensures high quality of responses to interactive user queries that are often not known apriori. To the best of our knowledge, iRAG is the first system to augment RAG with an incremental workflow to support efficient interactive querying of a large corpus of videos. Experimental results on real-world datasets demonstrate 23x to 25x faster video to text ingestion, while ensuring that latency and quality of responses to interactive user queries is comparable to responses from a traditional RAG where all video data is converted to text upfront before any user querying.