mlsanityData to 7 Oct 2026

Retrieval-Augmented Multimodal Language Modeling

arXiv 2211.12561 ↗First posted 22 Nov 2022cs.CV cs.CL cs.LGcited by 125

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in text-to-image and image-to-text generation. However, these models store all learned knowledge (e.g., the appearance of the Eiffel Tower) in the model parameters, requiring increasingly larger models and training data to capture more knowledge. To integrate knowledge in a more scalable and modular way, we propose a retrieval-augmented multimodal model, which enables a base multimodal model (generator) to refer to relevant text and images fetched by a retriever from external memory (e.g., documents on the web). Specifically, for the retriever, we use a pretrained CLIP, and for the generator, we train a CM3 Transformer on the LAION dataset. Our resulting model, named Retrieval-Augmented CM3 (RA-CM3), is the first multimodal model that can retrieve and generate both text and images. We show that RA-CM3 significantly outperforms baseline multimodal models such as DALL-E and CM3 on both image and caption generation tasks (12 FID and 17 CIDEr improvements on MS-COCO), while requiring much less compute for training (<30% of DALL-E). Moreover, we show that RA-CM3 exhibits novel capabilities, such as faithful image generation and multimodal in-context learning (e.g., image generation from demonstrations).

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 62 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

49 of 62 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
37 more
  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  13. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  14. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  15. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  16. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  17. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  18. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  19. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  20. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  21. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  22. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  23. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  24. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  25. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  26. 2019Controllable Text-to-Image Generationcited by 187
  27. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  28. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  29. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  30. 2022Language Models are General-Purpose Interfacescited by 96
  31. 2020Pre-training via Paraphrasingcited by 92
  32. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  33. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  34. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58
  35. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  36. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  37. 2021Cross-Modal Retrieval Augmentation for Multi-Modal Classificationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 36; 2024: 45; 2025: 27; 2026: 16.

125 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  3. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  4. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  5. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  6. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  7. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  8. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  9. 2023Generating Images with Multimodal Language Modelscited by 259
  10. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  11. 2024Generative Representational Instruction Tuningcited by 227
  12. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
38 more
  1. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  2. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  3. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  4. 2023Large Language Models as Analogical Reasonerscited by 107
  5. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  6. 2024Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Modelscited by 91
  7. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  8. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  9. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  10. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  11. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  12. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  13. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directionscited by 69
  14. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  15. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  16. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  17. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
  18. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  19. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  20. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  21. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  22. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  23. 2024Autoregressive Models in Vision: A Surveycited by 39
  24. 2023Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Modelscited by 37
  25. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  26. 2023Sieve: Multimodal Dataset Pruning Using Image Captioning Modelscited by 33
  27. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32
  28. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  29. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  30. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  31. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
  32. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  33. 2024Learning on Multimodal Graphs: A Surveycited by 22
  34. 2024Retrieval Augmented End-to-End Spoken Dialog Modelscited by 20
  35. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20
  36. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  37. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  38. 2025Graph4MM: Weaving Multimodal Learning with Structural Informationcited by 17

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  2. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant
  3. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
  4. 2024RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training
  5. 2025Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems
  6. 2025M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generation
  7. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?
  8. 2024Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs