mlsanityData to 13 Oct 2026

REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory

arXiv 2212.05221 ↗First posted 10 Dec 2022cs.CV cs.AIcited by 122

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

In this paper, we propose an end-to-end Retrieval-Augmented Visual Language Model (REVEAL) that learns to encode world knowledge into a large-scale memory, and to retrieve from it to answer knowledge-intensive queries. REVEAL consists of four key components: the memory, the encoder, the retriever and the generator. The large-scale memory encodes various sources of multimodal world knowledge (e.g. image-text pairs, question answering pairs, knowledge graph triplets, etc) via a unified encoder. The retriever finds the most relevant knowledge entries in the memory, and the generator fuses the retrieved knowledge with the input query to produce the output. A key novelty in our approach is that the memory, encoder, retriever and generator are all pre-trained end-to-end on a massive amount of data. Furthermore, our approach can use a diverse set of multimodal knowledge sources, which is shown to result in significant gains. We show that REVEAL achieves state-of-the-art results on visual question answering and image captioning.

Key facts. Cited by 122 papers in the map (data through 13 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 56 references that are in the map. First posted on arXiv 10 Dec 2022.

Builds on

46 of 56 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,797
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  10. 2015VQA: Visual Question Answeringcited by 2,495
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,131
34 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,035
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,551
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,452
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,267
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,211
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,116
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,094
  8. 2021Scaling Vision Transformerscited by 998
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 947
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 764
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 706
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 573
  14. 2021ClipCap: CLIP Prefix for Image Captioningcited by 558
  15. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  16. 2018nocaps: novel object captioning at scalecited by 444
  17. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  18. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  19. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  20. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  21. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  22. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  23. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  24. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  25. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  26. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  27. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  28. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  29. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  30. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  31. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  32. 2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
  33. 2022TPU-KNN: K Nearest Neighbor Search at Peak FLOP/scited by 18
  34. 2022Empowering Language Models with Knowledge Graph Reasoning for Question Answeringcited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 21; 2024: 41; 2025: 39; 2026: 21.

122 papers.

Sort:

By area:

  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 596
  2. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  3. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 353
  4. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  5. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 241
  6. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 205
  7. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  8. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 121
  9. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  10. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 106
  11. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  12. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
38 more
  1. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  2. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  3. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  4. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  5. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  6. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  7. 2024Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphscited by 38
  8. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  9. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  10. 2023Vocabulary-free Image Classificationcited by 35
  11. 2023Sieve: Multimodal Dataset Pruning Using Image Captioning Modelscited by 33
  12. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  13. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  14. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  15. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  16. 2024SciMMIR: Benchmarking Scientific Multi-modal Information Retrievalcited by 23
  17. 2024On the Evaluation of Machine-Generated Reportscited by 23
  18. 2024Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generationcited by 23
  19. 2025A Survey on Agentic Multimodal Large Language Modelscited by 22
  20. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  21. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  22. 2023Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Modelscited by 18
  23. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 18
  24. 2023Retrieval-based Knowledge Augmented Vision Language Pre-trainingcited by 17
  25. 2023ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Sciencecited by 15
  26. 2025SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generationcited by 15
  27. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14
  28. 2024A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysiscited by 14
  29. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Enginescited by 13
  30. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  31. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contextscited by 12
  32. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learningcited by 10
  33. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answeringcited by 9
  34. 2024Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Usecited by 9
  35. 2024Knowledge-aware Text-Image Retrieval for Remote Sensing Imagescited by 9
  36. 2024Retrieval Augmented Diffusion Model for Structure-informed Antibody Design and Optimizationcited by 9
  37. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  38. 2023AVIS: Autonomous Visual Information Seeking with Large Language Model Agentcited by 8

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
  2. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
  3. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  4. 2023Generating Images with Multimodal Language Models
  5. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
  6. 2026Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
  7. 2024Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs
  8. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering