mlsanityData to 7 Oct 2026

Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning

arXiv 2302.04858 ↗First posted 9 Feb 2023cs.CV cs.AI cs.CLcited by 41

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Augmenting pretrained language models (LMs) with a vision encoder (e.g., Flamingo) has obtained the state-of-the-art results in image-to-text generation. However, these models store all the knowledge within their parameters, thus often requiring enormous model parameters to model the abundant visual concepts and very rich textual descriptions. Additionally, they are inefficient in incorporating new data, requiring a computational-expensive fine-tuning process. In this work, we introduce a Retrieval-augmented Visual Language Model, Re-ViLM, built upon the Flamingo, that supports retrieving the relevant knowledge from the external database for zero and in-context few-shot image-to-text generations. By storing certain knowledge explicitly in the external database, our approach reduces the number of model parameters and can easily accommodate new data during evaluation by simply updating the database. We also construct an interleaved image and text data that facilitates in-context few-shot learning capabilities. We demonstrate that Re-ViLM significantly boosts performance for image-to-text generation tasks, especially for zero-shot and few-shot generation in out-of-domain settings with 4 times less parameters compared with baseline methods.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 45 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

35 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  3. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
19 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  3. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  7. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  9. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  10. 2018nocaps: novel object captioning at scalecited by 444
  11. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  12. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  13. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  14. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  15. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  16. 2018Neural Baby Talkcited by 107
  17. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  18. 2022Language Models are General-Purpose Interfacescited by 96
  19. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 19; 2025: 9; 2026: 4.

41 papers.

Sort:

By area:

  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  2. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  3. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  4. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  5. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  6. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  7. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 21
  8. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  9. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  10. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  11. 2024Altogether: Image Captioning via Re-aligning Alt-textcited by 17
  12. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
29 more
  1. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  2. 2024REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environmentscited by 12
  3. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Informationcited by 11
  4. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learningcited by 10
  5. 2023NICE: CVPR 2023 Challenge on Zero-shot Image Captioningcited by 9
  6. 2024Understanding Retrieval Robustness for Retrieval-Augmented Image Captioningcited by 9
  7. 2024Audio Dialogues: Dialogues dataset for audio and music understandingcited by 7
  8. 2023Retrieval-Enhanced Visual Prompt Learning for Few-shot Classificationcited by 6
  9. 2024CaMML: Context-Aware Multimodal Learner for Large Modelscited by 6
  10. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottleneckscited by 4
  11. 2024Survey of User Interface Design and Interaction Techniques in Generative AI Applicationscited by 3
  12. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Modelscited by 3
  13. 2025Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detectioncited by 3
  14. 2024Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tagscited by 2
  15. 2025Can Vision-Language Models Solve Visual Math Equations?cited by 2
  16. 2026Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Modelscited by 2
  17. 2023The Solution for the CVPR2023 NICE Image Captioning Challengecited by 1
  18. 2024DIR: Retrieval-Augmented Image Captioning with Comprehensive Understandingcited by 1
  19. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modelingcited by 1
  20. 2025Multimodal RAG Enhanced Visual Descriptioncited by 1
  21. 2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generationcited by 1
  22. 2025PoultryTalk: A Multi-modal Retrieval-Augmented Generation (RAG) System for Intelligent Poultry Management and Decision Supportcited by 1
  23. 2026ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMscited by 1
  24. 2026Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAGcited by 1
  25. 2023Linear Alignment of Vision-language Models for Image Captioningcited by 0
  26. 2023Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generationcited by 0
  27. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  28. 2025NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Taskscited by 0
  29. 2026Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gapcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023MLLMs-Augmented Visual-Language Representation Learning
  2. 2023Retrieval-augmented Image Captioning
  3. 2023Revisiting the Role of Language Priors in Vision-Language Models
  4. 2024Towards Retrieval-Augmented Architectures for Image Captioning
  5. 2025SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models
  6. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  7. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Models
  8. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions