mlsanityData to 14 Oct 2026

AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding

arXiv 2506.13589 ↗First posted 16 Jun 2025cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) perform well in video understanding but degrade on long videos due to fixed-length context and weak long-term dependency modeling. Retrieval-Augmented Generation (RAG) can expand knowledge dynamically, yet existing video RAG schemes adopt fixed retrieval paradigms that ignore query difficulty. This uniform design causes redundant computation and latency for simple queries, while coarse retrieval for complex, multi-hop reasoning can miss key information. Such single-step retrieval severely limits the trade-off between efficiency and cognitive depth. We propose AdaVideoRAG, an adaptive RAG framework for long-video understanding. A lightweight intent classifier dynamically selects suitable retrieval schemes according to query complexity from the simplest to the most sophisticated. We design an Omni-Knowledge Indexing module that extracts and organizes multi-modal information into three databases: (1) a text base built from clip captions, ASR, and OCR; (2) a visual base; and (3) a knowledge graph for deep semantic understanding. This supports hierarchical knowledge access, from naive retrieval to graph-based retrieval, balancing resource cost and reasoning ability. To evaluate deep understanding, we further construct the HiVU benchmark. Experiments show that AdaVideoRAG significantly improves both efficiency and accuracy on long-video QA tasks and can be seamlessly plugged into existing MLLMs through lightweight APIs, establishing a new paradigm for adaptive retrieval-augmented video analysis.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 54 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

47 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers14 Mar
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  4. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding18 Mar
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
29 more
  1. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  4. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  5. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  6. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  7. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  8. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  9. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  10. 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 315
  11. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  12. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
  13. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  14. 2024Corrective Retrieval Augmented Generationcited by 197
  15. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  16. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  17. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 91
  18. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  19. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  20. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 77
  21. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 71
  22. 2024TableRAG: Million-Token Table Understanding with Language Modelscited by 41
  23. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 40
  24. 2024OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquercited by 34
  25. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  26. 2025MiniRAG: Towards Extremely Simple Retrieval-Augmented Generationcited by 28
  27. 2024AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipelinecited by 27
  28. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  29. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Modelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 53
  2. 2025UltraGen: High-Resolution Video Generation with Hierarchical Attentioncited by 18
  3. 2026Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Searchcited by 8
  4. 2026Video-Oasis: Rethinking Evaluation of Video Understandingcited by 3
  5. 2026SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memorycited by 3
  6. 2026Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoningcited by 2
  7. 2026LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videoscited by 2
  8. 2026Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understandingcited by 2
  9. 2026Rethinking RAG in Long Videos: What to Retrieve and How to Use It?cited by 1
  10. 2025ALIVE: An Avatar-Lecture Interactive Video Engine with Content-Aware Retrieval for Real-Time Interactioncited by 0
  11. 2026Multi-Dimensional Knowledge Profiling with Large-Scale Literature Database and Hierarchical Retrievalcited by 0
  12. 2026QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understandingcited by 0
3 more
  1. 2026Agentic Video Understanding: A Surveycited by 0
  2. 2026LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perceptioncited by 0
  3. 2026EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAG
  2. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding
  3. 2025E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generation
  4. 2025TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding
  5. 2026Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation
  6. 2025SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding
  7. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  8. 2026LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension