mlsanityData to 14 Oct 2026

MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

arXiv 2210.02928 ↗First posted 6 Oct 2022cs.CL cs.AI cs.CVcited by 220

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs. Recently, retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated world knowledge into language generation by leveraging an external non-parametric index and have demonstrated impressive performance with constrained model sizes. However, these methods are restricted to retrieving only textual knowledge, neglecting the ubiquitous amount of knowledge in other modalities like images -- much of which contains information not covered by any text. To address this limitation, we propose the first Multimodal Retrieval-Augmented Transformer (MuRAG), which accesses an external non-parametric multimodal memory to augment language generation. MuRAG is pre-trained with a mixture of large-scale image-text and text-only corpora using a joint contrastive and generative loss. We perform experiments on two different datasets that require retrieving and reasoning over both images and text to answer a given query: WebQA, and MultimodalQA. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20% absolute on both datasets and under both distractor and full-wiki settings.

Key facts. Cited by 220 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 33 of its 42 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

33 of 42 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
21 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  6. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  13. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  14. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 200
  15. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  16. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  17. 2021WebQA: Multihop and Multimodal QAcited by 109
  18. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 43
  19. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38
  20. 2021MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Groundingcited by 19
  21. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answeringcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 22; 2024: 63; 2025: 70; 2026: 63.

220 papers.

Sort:

By area:

  1. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  2. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  3. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  4. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  5. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
  6. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  7. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  8. 2024CRAG -- Comprehensive RAG Benchmarkcited by 97
  9. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  10. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  11. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  12. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
38 more
  1. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  2. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  3. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  4. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 57
  5. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 55
  6. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 54
  7. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  8. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 51
  9. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  10. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  11. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  12. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 42
  13. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  14. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  15. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 41
  16. 2024Goldfish: Vision-Language Understanding of Arbitrarily Long Videoscited by 40
  17. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  18. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  19. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 36
  20. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 36
  21. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  22. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applicationscited by 31
  23. 2022Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generationcited by 28
  24. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  25. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 28
  26. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defectscited by 27
  27. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  28. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  29. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  30. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  31. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  32. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  33. 2024SciMMIR: Benchmarking Scientific Multi-modal Information Retrievalcited by 23
  34. 2024Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generationcited by 23
  35. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  36. 2023Unified Language Representation for Question Answering over Text, Tables, and Imagescited by 22
  37. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 21
  38. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20

50 most cited of 220 All 220 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Taming a Retrieval Framework to Read Images in Humanlike Manner for Augmenting Generation of MLLMs
  2. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
  3. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
  4. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
  5. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
  6. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  7. 2025RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering
  8. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs