mlsanityData to 14 Oct 2026

Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

arXiv 2502.08826 ↗First posted 12 Feb 2025cs.CL cs.AI cs.IRcited by 42

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges beyond those in unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We review training strategies, robustness enhancements, loss functions, and agent-based approaches, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. All resources are publicly available at https://github.com/llm-lab-org/Multimodal-RAG-Survey.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 194 of its 303 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

194 of 303 references.

Closest ancestors · 25

Same direction, newest first

  1. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos3 Feb
  2. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpus10 Jan
  3. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance7 Jan
  4. 2025Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning2 Jan
  5. 2024VISA: Retrieval Augmented Generation with Visual Source Attribution19 Dec
  6. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation14 Dec
  7. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations10 Dec
  8. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension20 Nov
  9. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding7 Nov
  10. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent5 Nov
  11. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications29 Oct
  12. 2024RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training18 Oct
13 more
  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents14 Oct
  2. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning12 Oct
  3. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models10 Oct
  4. 2024MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering16 Aug
  5. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training31 Jul
  6. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs28 Jun
  7. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learning27 Jun
  8. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr
  9. 2024iRAG: Advancing RAG for Videos with an Incremental Approach18 Apr
  10. 2024MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning21 Feb
  11. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
  12. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct
  13. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generator29 Sep

Other references · 169

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2024The Llama 3 Herd of Modelscited by 14,715
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  13. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  14. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  15. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  16. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  17. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  18. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  19. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  20. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  21. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  22. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  23. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  24. 2018Demystifying MMD GANscited by 1,293
  25. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  26. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  27. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  28. 2023PaLM 2 Technical Reportcited by 1,162
  29. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  30. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  31. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  32. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  33. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  34. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  35. 2017Dense-Captioning Events in Videoscited by 793
  36. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  37. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  38. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719

50 most cited of 169 All 169 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 18.

42 papers.

Sort:

By area:

  1. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agentscited by 51
  2. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27
  3. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 20
  4. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 20
  5. 2025RAG-Anything: All-in-One RAG Frameworkcited by 20
  6. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 18
  7. 2025CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoningcited by 11
  8. 2026From Rows to Reasoning: A Retrieval-Augmented Multimodal Framework for Spreadsheet Understandingcited by 6
  9. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  10. 2025Privacy-Preserving LLM Embedding Transmission for End-Cloud Collaborationcited by 5
  11. 2025Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systemscited by 5
  12. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
30 more
  1. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrievalcited by 4
  2. 2025ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoningcited by 3
  3. 2025MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessmentcited by 3
  4. 2025HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generationcited by 3
  5. 2026CrisiSense-RAG: Crisis Sensing Multimodal Retrieval-Augmented Generation for Rapid Disaster Impact Assessmentcited by 3
  6. 2025VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generationcited by 2
  7. 2025Learning to Retrieve for Environmental Knowledge Discovery: An Augmentation-Adaptive Self-Supervised Learning Frameworkcited by 2
  8. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrievalcited by 2
  9. 2026MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrievalcited by 2
  10. 2026MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAGcited by 2
  11. 2025A Visual RAG Pipeline for Few-Shot Fine-Grained Product Classificationcited by 1
  12. 2025A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answeringcited by 1
  13. 2025EBind: a practical approach to space bindingcited by 1
  14. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAGcited by 1
  15. 2026RefereeBench: Are Video MLLMs Ready to be Multi-Sport Refereescited by 1
  16. 2026MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&Acited by 1
  17. 2026Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAGcited by 1
  18. 2026Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphscited by 1
  19. 2025Agent-based Condition Monitoring Assistance with Multimodal Industrial Database Retrieval Augmented Generationcited by 0
  20. 2025Evaluating VisualRAG: Quantifying Cross-Modal Performance in Enterprise Document Understandingcited by 0
  21. 2025Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuningcited by 0
  22. 2025Chunking Strategies for Multimodal AI Systemscited by 0
  23. 2026Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generationcited by 0
  24. 2026E-MMKGR: A Unified Multimodal Knowledge Graph Framework for E-commerce Applicationscited by 0
  25. 2026Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generationcited by 0
  26. 2026Citation-Enforced RAG for Fiscal Document Intelligence: Cited, Explainable Knowledge Retrieval in Tax Compliancecited by 0
  27. 2026FollowTable: A Benchmark for Instruction-Following Table Retrievalcited by 0
  28. 2026From Naive RAG to Deep Agentic Retrieval: An Evolving Context Engineering Pipeline for Regulatory Compliancecited by 0
  29. 2026KoViDoRe: A Benchmark for Korean Visual Document Retrievalcited by 0
  30. 2026Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  2. 2024Searching for Best Practices in Retrieval-Augmented Generation
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
  4. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
  5. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts
  6. 2025FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
  7. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generation
  8. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation