mlsanityData to 14 Oct 2026

A Survey of Multimodal Retrieval-Augmented Generation

arXiv 2504.08748 ↗First posted 26 Mar 2025cs.IR cs.AI cs.CLcited by 37

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Multimodal Retrieval-Augmented Generation (MRAG) enhances large language models (LLMs) by integrating multimodal data (text, images, videos) into retrieval and generation processes, overcoming the limitations of text-only Retrieval-Augmented Generation (RAG). While RAG improves response accuracy by incorporating external textual knowledge, MRAG extends this framework to include multimodal retrieval and generation, leveraging contextual information from diverse data types. This approach reduces hallucinations and enhances question-answering systems by grounding responses in factual, multimodal knowledge. Recent studies show MRAG outperforms traditional RAG, especially in scenarios requiring both visual and textual understanding. This survey reviews MRAG's essential components, datasets, evaluation methods, and limitations, providing insights into its construction and improvement. It also identifies challenges and future research directions, highlighting MRAG's potential to revolutionize multimodal information retrieval and generation. By offering a comprehensive perspective, this work encourages further exploration into this promising paradigm.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 265 of its 514 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

265 of 514 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation6 Feb
  2. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation14 Dec
  3. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation3 Dec
  4. 2024Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines25 Nov
  5. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent5 Nov
  6. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications29 Oct
  7. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents14 Oct
  8. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models10 Oct
  9. 2024MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering16 Aug
  10. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep

Other references · 255

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
38 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  5. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  15. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  16. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  17. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  19. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  20. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  21. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  22. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  23. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  24. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  25. 2019Passage Re-ranking with BERTcited by 645
  26. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  27. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  28. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  29. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  30. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  31. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  32. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  33. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  34. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  35. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  36. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  37. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  38. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370

50 most cited of 255 All 255 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 20.

37 papers.

Sort:

By area:

  1. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  2. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 31
  3. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  4. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 20
  5. 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Modelscited by 19
  6. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 15
  7. 2026Towards Long-horizon Agentic Multimodal Searchcited by 11
  8. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  9. 2025AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learningcited by 8
  10. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  11. 2025M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generationcited by 7
  12. 2025A Comprehensive Survey of Knowledge-Based Vision Question Answering Systems: The Lifecycle of Knowledge in Visual Reasoning Taskcited by 6
25 more
  1. 2026From Rows to Reasoning: A Retrieval-Augmented Multimodal Framework for Spreadsheet Understandingcited by 6
  2. 2025Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systemscited by 5
  3. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  4. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrievalcited by 4
  5. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 4
  6. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 4
  7. 2025SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controllingcited by 3
  8. 2025HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generationcited by 3
  9. 2026Multimodal Adaptive Retrieval Augmented Generation through Internal Representation Learningcited by 3
  10. 2026SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Taskcited by 3
  11. 2026MultiPress: A Multi-Agent Framework for Interpretable Multimodal News Classificationcited by 2
  12. 2026Automotive Engineering-Centric Agentic AI Workflow Frameworkcited by 2
  13. 2026Very Efficient Listwise Multimodal Reranking for Long Documentscited by 2
  14. 2025Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environmentcited by 1
  15. 2026Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generationcited by 1
  16. 2026MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&Acited by 1
  17. 2025In-Context-Learning-Assisted Quality Assessment Vision-Language Models for Metal Additive Manufacturingcited by 0
  18. 2025Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuningcited by 0
  19. 2026Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generationcited by 0
  20. 2026Explainable Innovation Engine: Dual-Tree Agent-RAG with Methods-as-Nodes and Verifiable Write-Backcited by 0
  21. 2026Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneitycited by 0
  22. 2026Evidence-Grounded Multimodal Knowledge Graph Construction for Multi-Lecture Educational Reasoningcited by 0
  23. 2026VDGR-RAG: Vectors, Directories, Graphs, and Reflection Are All You Need for Unified Reasoning over Hierarchical Enterprise Knowledgecited by 0
  24. 2026Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automationcited by 0
  25. 2026PACE: Publisher-Adaptive Content Extraction via Agentic Automationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
  2. 2024Searching for Best Practices in Retrieval-Augmented Generation
  3. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  4. 2024Retrieval-augmented generation in multilingual settings
  5. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  6. 2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation
  7. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering
  8. 2024Multi-Head RAG: Solving Multi-Aspect Problems with LLMs