mlsanityData to 7 Oct 2026

Understanding Retrieval Robustness for Retrieval-Augmented Image Captioning

arXiv 2406.02265 ↗First posted 4 Jun 2024cs.CV cs.CLcited by 9

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Recent advances in retrieval-augmented models for image captioning highlight the benefit of retrieving related captions for efficient, lightweight models with strong domain-transfer capabilities. While these models demonstrate the success of retrieval augmentation, retrieval models are still far from perfect in practice: the retrieved information can sometimes mislead the model, resulting in incorrect generation and worse performance. In this paper, we analyze the robustness of a retrieval-augmented captioning model SmallCap. Our analysis shows that the model is sensitive to tokens that appear in the majority of the retrieved captions, and the input attribution shows that those tokens are likely copied into the generated output. Given these findings, we propose to train the model by sampling retrieved captions from more diverse sets. This decreases the chance that the model learns to copy majority tokens, and improves both in-domain and cross-domain performance.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 26 of its 32 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

26 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension27 Nov
  2. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning9 Feb
  3. 2022Retrieval-Augmented Transformer for Image Captioning26 Jul

Other references · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2018nocaps: novel object captioning at scalecited by 444
11 more
  1. 2023Augmented Language Models: a Surveycited by 398
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  3. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  4. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  5. 2020Captioning Images Taken by People Who Are Blindcited by 108
  6. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  7. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 82
  8. 2023Style-Aware Contrastive Learning for Multi-Style Image Captioningcited by 21
  9. 2024RAGged Edges: The Double-Edged Sword of Retrieval-Augmented Chatbotscited by 11
  10. 2022Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactionscited by 7
  11. 2022Improving Robustness of Retrieval Augmented Translation via Shuffling of Suggestionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  2. 2025SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compressioncited by 9
  3. 2026QualiRAG: Retrieval-Augmented Generation for Visual Quality Understandingcited by 2
  4. 2025CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioningcited by 0
  5. 2025Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructionscited by 0
  6. 2025Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correctioncited by 0
  7. 2026Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioningcited by 0
  8. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Modelscited by 0
  9. 2026RA-CoA: Training-free Fashion Image Captioning via Retrieval-Augmented Chain-of-Attributescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Retrieval-Augmented Architectures for Image Captioning
  2. 2023Retrieval-augmented Image Captioning
  3. 2024Benchmark Granularity and Model Robustness for Image-Text Retrieval
  4. 2024DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding
  5. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
  6. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?
  7. 2024Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning
  8. 2025RACap: Relation-Aware Prompting for Lightweight Retrieval-Augmented Image Captioning