mlsanityData to 14 Oct 2026

EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension

arXiv 2311.15879 ↗First posted 27 Nov 2023cs.CVcited by 32

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Large language models (LLMs)-based image captioning has the capability of describing objects not explicitly observed in training data; yet novel objects occur frequently, necessitating the requirement of sustaining up-to-date object knowledge for open-world comprehension. Instead of relying on large amounts of data and/or scaling up network parameters, we introduce a highly effective retrieval-augmented image captioning method that prompts LLMs with object names retrieved from External Visual--name memory (EVCap). We build ever-changing object knowledge memory using objects' visuals and names, enabling us to (i) update the memory at a minimal cost and (ii) effortlessly augment LLMs with retrieved object names by utilizing a lightweight and fast-to-train model. Our model, which was trained only on the COCO dataset, can adapt to out-of-domain without requiring additional fine-tuning or re-training. Our experiments conducted on benchmarks and synthetic commonsense-violating data show that EVCap, with only 3.97M trainable parameters, exhibits superior performance compared to other methods based on frozen pre-trained LLMs. Its performance is also competitive to specialist SOTAs that require extensive training.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 46 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

38 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Retrieval-augmented Image Captioning16 Feb
  2. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning9 Feb
  3. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
  4. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov

Other references · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
22 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2017Billion-scale similarity search with GPUscited by 2,497
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  13. 2018nocaps: novel object captioning at scalecited by 444
  14. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  15. 2018Neural Baby Talkcited by 107
  16. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  17. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  18. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  19. 2015Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Datacited by 36
  20. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  21. 2016Captioning Images with Diverse Objectscited by 27
  22. 2022NOC-REK: Novel Object Captioning with Retrieved Vocabulary from External Knowledgecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 12; 2026: 9.

32 papers.

Sort:

By area:

  1. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  2. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  3. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 27
  4. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  5. 2025FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowchartscited by 18
  6. 2025Domain-RAG: Retrieval-Guided Compositional Image Generation for Cross-Domain Few-Shot Object Detectioncited by 15
  7. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucinationcited by 14
  8. 2024R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generationcited by 12
  9. 2024Understanding Retrieval Robustness for Retrieval-Augmented Image Captioningcited by 9
  10. 2024From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichmentcited by 8
  11. 2024Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Modelcited by 8
  12. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLMcited by 6
20 more
  1. 2025RA-RRG: Multimodal Retrieval-Augmented Radiology Report Generation with Key Phrase Extractioncited by 4
  2. 2025Event-Enriched Image Analysis Grand Challenge at ACM Multimedia 2025cited by 3
  3. 2025Contextual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilitiescited by 3
  4. 2026Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generationcited by 3
  5. 2026Models Designed to Forget: Machine Unlearning via Key Deletioncited by 3
  6. 2024Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tagscited by 2
  7. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1
  8. 2024DIR: Retrieval-Augmented Image Captioning with Comprehensive Understandingcited by 1
  9. 2025The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioningcited by 0
  10. 2025Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Modelcited by 0
  11. 2025CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioningcited by 0
  12. 2025RACap: Relation-Aware Prompting for Lightweight Retrieval-Augmented Image Captioningcited by 0
  13. 2025DualCap: Enhancing Lightweight Image Captioning via Dual Retrieval with Similar Scenes Visual Promptscited by 0
  14. 2026Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learningcited by 0
  15. 2026ProCap: Projection-Aware Captioning for Spatial Augmented Realitycited by 0
  16. 2026VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detectioncited by 0
  17. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Modelscited by 0
  18. 2026Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioningcited by 0
  19. 2026Re3Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learningcited by 0
  20. 2026RA-CoA: Training-free Fashion Image Captioning via Retrieval-Augmented Chain-of-Attributescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Retrieval-Augmented Architectures for Image Captioning
  2. 2022Retrieval-Augmented Transformer for Image Captioning
  3. 2024Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes
  4. 2024FlexCap: Describe Anything in Images in Controllable Detail
  5. 2022PromptCap: Prompt-Guided Task-Aware Image Captioning
  6. 2024D'ej`a Vu Memorization in Vision-Language Models
  7. 2026Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning
  8. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learning