mlsanityData to 14 Oct 2026

Retrieval-Augmented Transformer for Image Captioning

arXiv 2207.13162 ↗First posted 26 Jul 2022cs.CV cs.AI cs.CLcited by 38

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Image captioning models aim at connecting Vision and Language by providing natural language descriptions of input images. In the past few years, the task has been tackled by learning parametric models and proposing visual feature extraction advancements or by modeling better multi-modal connections. In this paper, we investigate the development of an image captioning approach with a kNN memory, with which knowledge can be retrieved from an external corpus to aid the generation process. Our architecture combines a knowledge retriever based on visual similarities, a differentiable encoder, and a kNN-augmented attention layer to predict tokens based on the past context and on text retrieved from the external memory. Experimental results, conducted on the COCO dataset, demonstrate that employing an explicit external memory can aid the generation process and increase caption quality. Our work opens up new avenues for improving image captioning models at larger scale.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 56 of its 65 references that are in the map. First posted on arXiv 26 Jul 2022.

Builds on

56 of 65 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2020Language Models are Few-Shot Learnerscited by 32,419
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2017Billion-scale similarity search with GPUscited by 2,497
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  10. 2017Mixed Precision Trainingcited by 1,156
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  13. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  14. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  15. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  16. 2016Self-critical Sequence Training for Image Captioningcited by 547
  17. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  18. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  19. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  20. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  21. 2019Attention on Attention for Image Captioningcited by 289
  22. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  23. 2022Memorizing Transformerscited by 226
  24. 2018Exploring Visual Relationship for Image Captioningcited by 215
  25. 2020X-Linear Attention Networks for Image Captioningcited by 198
  26. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  27. 2019Image Captioning: Transforming Objects into Wordscited by 171
  28. 2015Particular object retrieval with integral max-pooling of CNN activationscited by 161
  29. 2019Augmenting Self-attention with Persistent Memorycited by 93
  30. 2021Training Vision Transformers for Image Retrievalcited by 90
  31. 2020Improving Image Captioning with Better Use of Captionscited by 84
  32. 2021Adaptive Semiparametric Language Modelscited by 83
  33. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  34. 2020Normalized and Geometry-Aware Self-Attention Network for Image Captioningcited by 63
  35. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  36. 2017Convolutional Image Captioningcited by 52
  37. 2019Learning to Collocate Neural Modules for Image Captioningcited by 22
  38. 2020Explore and Explain: Self-supervised Navigation and Recountingcited by 14

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 16; 2025: 6; 2026: 2.

38 papers.

Sort:

By area:

  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  3. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  4. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  5. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  6. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  7. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  8. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  9. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  10. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  11. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  12. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
26 more
  1. 2023Retrieval-augmented Image Captioningcited by 29
  2. 2023Parents and Children: Distinguishing Multimodal DeepFakes from Natural Imagescited by 29
  3. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  4. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  5. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 27
  6. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18
  7. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  8. 2024Accelerating Retrieval-Augmented Generationcited by 14
  9. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  10. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learningcited by 10
  11. 2024Understanding Retrieval Robustness for Retrieval-Augmented Image Captioningcited by 9
  12. 2024VIMI: Grounding Video Generation through Multi-modal Instructioncited by 8
  13. 2024Towards Retrieval-Augmented Architectures for Image Captioningcited by 7
  14. 2024Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMscited by 6
  15. 2025CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generationcited by 6
  16. 2025Multispectral Pedestrian Detection with Sparsely Annotated Labelcited by 5
  17. 2025RA-RRG: Multimodal Retrieval-Augmented Radiology Report Generation with Key Phrase Extractioncited by 4
  18. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Boundscited by 4
  19. 2022Vision and Structured-Language Pretraining for Cross-Modal Food Retrievalcited by 3
  20. 2025TADACap: Time-series Adaptive Domain-Aware Captioningcited by 3
  21. 2024Composing Open-domain Vision with RAG for Ocean Monitoring and Conservationcited by 1
  22. 2025Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correctioncited by 1
  23. 2026SAR-RAG: ATR Visual Question Answering by Semantic Search, Retrieval, and MLLM Generationcited by 1
  24. 2023Learning the Visualness of Text Using Large Vision-Language Modelscited by 0
  25. 2024Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answeringcited by 0
  26. 2026Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension
  2. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
  3. 2025Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation
  4. 2024ViTOC: Vision Transformer and Object-aware Captioner
  5. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
  6. 2025Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model
  7. 2022Deep Learning Approaches on Image Captioning: A Review
  8. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory