mlsanityData to 7 Oct 2026

Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension

arXiv 2411.13093 ↗First posted 20 Nov 2024cs.CV cs.AIcited by 134

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 41 of its 54 references that are in the map. First posted on arXiv 20 Nov 2024.

Builds on

41 of 54 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
29 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  5. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  8. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  9. 2024Long Context Transfer from Language to Visioncited by 502
  10. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  11. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  12. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  13. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  14. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  15. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  16. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  17. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  18. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  19. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  20. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  21. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  22. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  23. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  24. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  25. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  26. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  27. 2023Aligning and Prompting Everything All at Once for Universal Visual Perceptioncited by 59
  28. 2024OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquercited by 34
  29. 2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 66; 2026: 67.

134 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  3. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  4. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  5. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  6. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
  7. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  8. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  9. 2025Kwai Keye-VL Technical Reportcited by 47
  10. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  11. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  12. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
38 more
  1. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  2. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  3. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  4. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  5. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 30
  6. 2025FilMaster: Bridging Cinematic Principles and Generative AI for Automated Film Generationcited by 28
  7. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  8. 2025LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoningcited by 27
  9. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  10. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  11. 2025LongVideoAgent: Multi-Agent Reasoning with Long Videoscited by 23
  12. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  13. 2025Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?cited by 19
  14. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19
  15. 2026Flow caching for autoregressive video generationcited by 19
  16. 2025Memory-enhanced Retrieval Augmentation for Long Video Understandingcited by 17
  17. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  18. 2026OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoningcited by 16
  19. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  20. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understandingcited by 14
  21. 2026From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agentscited by 14
  22. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 14
  23. 2025Multi-RAG: A Multimodal Retrieval-Augmented Generation System for Adaptive Video Understandingcited by 13
  24. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Modelscited by 13
  25. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 13
  26. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 12
  27. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  28. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  29. 2025E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generationcited by 11
  30. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  31. 2025Video-EM: Event-Centric Episodic Memory for Long-Form Video Understandingcited by 10
  32. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  33. 2025RAIDX: A Retrieval-Augmented Generation and GRPO Reinforcement Learning Framework for Explainable Deepfake Detectioncited by 8
  34. 2026Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Searchcited by 8
  35. 2026Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understandingcited by 8
  36. 2026Symphony: A Cognitively-Inspired Multi-Agent System for Long-Video Understandingcited by 8
  37. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  38. 2025Learning Plug-and-play Memory for Guiding Video Diffusion Modelscited by 7

50 most cited of 134 All 134 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Video Enriched Retrieval Augmented Generation Using Aligned Video Captions
  2. 2026LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension
  3. 2026Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation
  4. 2025RAG-Adapter: A Plug-and-Play RAG-enhanced Framework for Long Video Understanding
  5. 2026FastV-RAG: Towards Fast and Fine-Grained Video QA with Retrieval-Augmented Generation
  6. 2026EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding
  7. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  8. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation