mlsanityData to 7 Oct 2026

Retrieving Multimodal Information for Augmented Generation: A Survey

arXiv 2303.10868 ↗First posted 20 Mar 2023cs.CLcited by 84

Not on the map. Why?

Cited by 41 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As Large Language Models (LLMs) become popular, there emerged an important trend of using multimodality to augment the LLMs' generation ability, which enables LLMs to better interact with the world. However, there lacks a unified perception of at which stage and how to incorporate different modalities. In this survey, we review methods that assist and augment generative models by retrieving multimodal knowledge, whose formats range from images, codes, tables, graphs, to audio. Such methods offer a promising solution to important concerns such as factuality, reasoning, interpretability, and robustness. By providing an in-depth review, this survey is expected to provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 177 of its 256 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

177 of 256 references.

References · 177

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  13. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  14. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  15. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  16. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  17. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  18. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  19. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  20. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  21. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  22. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  23. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  24. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  25. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  26. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  27. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  28. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  29. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  30. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  31. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  32. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  33. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  34. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  35. 2022PAL: Program-aided Language Modelscited by 635
  36. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  37. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  38. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513

50 most cited of 177 All 177 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 37; 2025: 28; 2026: 14.

84 papers.

Sort:

By area:

  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  3. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  4. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  5. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  6. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 121
  7. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  8. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  9. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
  10. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  11. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
  12. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
38 more
  1. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  2. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  3. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  4. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  5. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  6. 2023PDFTriage: Question Answering over Long, Structured Documentscited by 29
  7. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  8. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  9. 2024Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answeringcited by 25
  10. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  11. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  12. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  13. 2024SFR-RAG: Towards Contextually Faithful LLMscited by 19
  14. 2024The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Modelscited by 17
  15. 2024Survey of Large Multimodal Model Datasets, Application Categories and Taxonomycited by 17
  16. 2025Can Multimodal Large Language Models be Guided to Improve Industrial Anomaly Detection?cited by 17
  17. 2024Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decompositioncited by 12
  18. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  19. 2024ChatHuman: Chatting about 3D Humans with Toolscited by 11
  20. 2024Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Stylecited by 11
  21. 2025HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insightscited by 11
  22. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answeringcited by 10
  23. 2024XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tagscited by 9
  24. 2025SuperRAG: Beyond RAG with Layout-Aware Graph Modelingcited by 9
  25. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  26. 2024Synthetic Multimodal Question Generationcited by 8
  27. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
  28. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  29. 2024PDF-MVQA: A Dataset for Multimodal Information Retrieval in PDF-based Visual Question Answeringcited by 7
  30. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Frameworkcited by 7
  31. 2025ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understandingcited by 7
  32. 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systemscited by 7
  33. 2024A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Casecited by 6
  34. 2025RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practicecited by 6
  35. 2025What Makes a Good Natural Language Prompt?cited by 6
  36. 2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddingscited by 5
  37. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  38. 2024How to Make LMs Strong Node Classifiers?cited by 4

50 most cited of 84 All 84 (JSON)