mlsanityData to 14 Oct 2026

Retrieval-augmented Image Captioning

arXiv 2302.08268 ↗First posted 16 Feb 2023cs.CV cs.CLcited by 29

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Inspired by retrieval-augmented language generation and pretrained Vision and Language (V&L) encoders, we present a new approach to image captioning that generates sentences given the input image and a set of captions retrieved from a datastore, as opposed to the image alone. The encoder in our model jointly processes the image and retrieved captions using a pretrained V&L BERT, while the decoder attends to the multimodal encoder representations, benefiting from the extra textual evidence from the retrieved captions. Experimental results on the COCO dataset show that image captioning can be effectively formulated from this new perspective. Our model, named EXTRA, benefits from using captions retrieved from the training dataset, and it can also benefit from using an external dataset without the need for retraining. Ablation studies show that retrieving a sufficient number of captions (e.g., k=5) can improve captioning quality. Our work contributes towards using pretrained V&L encoders for generative tasks, instead of standard classification tasks.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 43 of its 63 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

43 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Retrieval-Augmented Transformer for Image Captioning26 Jul

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2017Billion-scale similarity search with GPUscited by 2,497
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
30 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  8. 2021Ethical and social risks of harm from Language Modelscited by 966
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  12. 2016Self-critical Sequence Training for Image Captioningcited by 547
  13. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  14. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  15. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  16. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  17. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  18. 2019Attention on Attention for Image Captioningcited by 289
  19. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  20. 2018Exploring Visual Relationship for Image Captioningcited by 215
  21. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  22. 2019Image Captioning: Transforming Objects into Wordscited by 171
  23. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  24. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  25. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  26. 2018Retrieve and Refine: Improved Sequence Generation Models For Dialoguecited by 82
  27. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  28. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  29. 2020Show, Recall, and Tell: Image Captioning with Recall Mechanismcited by 27
  30. 2022CaMEL: Mean Teacher Learning for Image Captioningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 8; 2025: 8; 2026: 6.

29 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  3. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 59
  4. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  5. 2023Vocabulary-free Image Classificationcited by 35
  6. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  7. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  8. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 27
  9. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  10. 2024Retrieval-Augmented Open-Vocabulary Object Detectioncited by 20
  11. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  12. 2024Accelerating Retrieval-Augmented Generationcited by 14
17 more
  1. 2024Vocabulary-free Image Classification and Semantic Segmentationcited by 8
  2. 2025Event-Enriched Image Analysis Grand Challenge at ACM Multimedia 2025cited by 3
  3. 2026Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generationcited by 3
  4. 2023RIGHT: Retrieval-augmented Generation for Mainstream Hashtag Recommendationcited by 2
  5. 2025RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Datacited by 2
  6. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1
  7. 2024DIR: Retrieval-Augmented Image Captioning with Comprehensive Understandingcited by 1
  8. 2025Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correctioncited by 1
  9. 2023Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generationcited by 0
  10. 2025RORPCap: Retrieval-based Objects and Relations Prompt for Image Captioningcited by 0
  11. 2025DualCap: Enhancing Lightweight Image Captioning via Dual Retrieval with Similar Scenes Visual Promptscited by 0
  12. 2025Multilingual Training-Free Remote Sensing Image Captioningcited by 0
  13. 2026Multi-Modal LLM based Image Captioning in ICT: Bridging the Gap Between General and Industry Domaincited by 0
  14. 2026Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioningcited by 0
  15. 2026Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answeringcited by 0
  16. 2026RA-CoA: Training-free Fashion Image Captioning via Retrieval-Augmented Chain-of-Attributescited by 0
  17. 2026TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Retrieval-Augmented Architectures for Image Captioning
  2. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
  3. 2025CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning
  4. 2024Understanding Retrieval Robustness for Retrieval-Augmented Image Captioning
  5. 2023Image Captioners Are Scalable Vision Learners Too
  6. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
  7. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
  8. 2024Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval