mlsanityData to 14 Oct 2026

SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation

arXiv 2209.15323 ↗First posted 30 Sep 2022cs.CV cs.CLcited by 82

Not on the map. Why?

Cited by 25 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning. As an alternative to large models, we present SmallCap, which generates a caption conditioned on an input image and related captions retrieved from a datastore. Our model is lightweight and fast to train, as the only learned parameters are in newly introduced cross-attention layers between a pre-trained CLIP encoder and GPT-2 decoder. SmallCap can transfer to new domains without additional finetuning and can exploit large-scale data in a training-free fashion since the contents of the datastore can be readily replaced. Our experiments show that SmallCap, trained only on COCO, has competitive performance on this benchmark, and also transfers to other domains without retraining, solely through retrieval from target-domain data. Further improvement is achieved through the training-free exploitation of diverse human-labeled and web data, which proves to be effective for a range of domains, including the nocaps benchmark, designed to test generalization to unseen visual concepts.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

35 of 52 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2017Decoupled Weight Decay Regularizationcited by 1,831
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
23 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  3. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  8. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  10. 2018nocaps: novel object captioning at scalecited by 444
  11. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  12. 2019Clotho: An Audio Captioning Datasetcited by 358
  13. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  14. 2019Connecting Vision and Language with Localized Narrativescited by 226
  15. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  16. 2022A Survey on Retrieval-Augmented Text Generationcited by 158
  17. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  18. 2020Captioning Images Taken by People Who Are Blindcited by 110
  19. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  20. 2022Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Datacited by 61
  21. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  22. 2021ReFormer: The Relational Transformer for Image Captioningcited by 23
  23. 2022CaMEL: Mean Teacher Learning for Image Captioningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 18; 2024: 31; 2025: 25; 2026: 7.

82 papers.

Sort:

By area:

  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  3. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  4. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 86
  5. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  6. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  7. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 59
  8. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  9. 2023CLIP in Medical Imaging: A Surveycited by 40
  10. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attackscited by 39
  11. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 36
  12. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
38 more
  1. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  2. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  3. 2023Retrieval-augmented Image Captioningcited by 29
  4. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 29
  5. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  6. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  7. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  8. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 27
  9. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editingcited by 25
  10. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  11. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18
  12. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  13. 2024Altogether: Image Captioning via Re-aligning Alt-textcited by 17
  14. 2024Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learningcited by 16
  15. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  16. 2025Domain-RAG: Retrieval-Guided Compositional Image Generation for Cross-Domain Few-Shot Object Detectioncited by 15
  17. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  18. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  19. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Informationcited by 11
  20. 2024IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioningcited by 11
  21. 2023X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentationcited by 10
  22. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learningcited by 10
  23. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  24. 2024Understanding Retrieval Robustness for Retrieval-Augmented Image Captioningcited by 9
  25. 2024VIMI: Grounding Video Generation through Multi-modal Instructioncited by 8
  26. 2023Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioningcited by 7
  27. 2023MindGPT: Interpreting What You See with Non-invasive Brain Recordingscited by 6
  28. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 6
  29. 2023User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioningcited by 5
  30. 2024Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimizationcited by 5
  31. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  32. 2025ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interactioncited by 5
  33. 2025Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projectorcited by 5
  34. 2023Top-Down Framework for Weakly-supervised Grounded Image Captioningcited by 4
  35. 2024A Survey on Large Language Models from Concept to Implementationcited by 4
  36. 2025RA-RRG: Multimodal Retrieval-Augmented Radiology Report Generation with Key Phrase Extractioncited by 4
  37. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Boundscited by 4
  38. 2024PC2: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrievalcited by 3

50 most cited of 82 All 82 (JSON)