mlsanityData to 7 Oct 2026

LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting

arXiv 2305.19821 ↗First posted 31 May 2023cs.CL cs.CVcited by 17

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Multilingual image captioning has recently been tackled by training with large-scale machine translated data, which is an expensive, noisy, and time-consuming process. Without requiring any multilingual caption data, we propose LMCap, an image-blind few-shot multilingual captioning model that works by prompting a language model with retrieved captions. Specifically, instead of following the standard encoder-decoder paradigm, given an image, LMCap first retrieves the captions of similar images using a multilingual CLIP encoder. These captions are then combined into a prompt for an XGLM decoder, in order to generate captions in the desired language. In other words, the generation model does not directly process the image, instead processing retrieved captions. Experiments on the XM3600 dataset of geographically diverse images show that our model is competitive with fully-supervised multilingual captioning models, without requiring any supervised training on any captioning data.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 43 of its 68 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

43 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Retrieval-augmented Image Captioning16 Feb
  2. 2022Retrieval-Augmented Transformer for Image Captioning26 Jul

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
29 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2021Scaling Vision Transformerscited by 993
  3. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  8. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  13. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  14. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  15. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  16. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  17. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  18. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  19. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  20. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  21. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  22. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  23. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  24. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  25. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  26. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  27. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  28. 2018Unpaired Image Captioning by Language Pivotingcited by 27
  29. 2022Multilingual Multimodal Learning with Machine Translated Textcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 8; 2026: 1.

17 papers.

Sort:

By area:

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  2. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  3. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  4. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 27
  5. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  6. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Informationcited by 11
  7. 2024Multilingual Vision-Language Pre-training for the Remote Sensing Domaincited by 7
  8. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  9. 2025What do vision-language models see in the context? Investigating multimodal in-context learningcited by 4
  10. 2025Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-timecited by 1
  11. 2025Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioningcited by 1
  12. 2023Linear Alignment of Vision-language Models for Image Captioningcited by 0
5 more
  1. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  2. 2025CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioningcited by 0
  3. 2025DualCap: Enhancing Lightweight Image Captioning via Dual Retrieval with Similar Scenes Visual Promptscited by 0
  4. 2025Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correctioncited by 0
  5. 2026MUNIChus: Multilingual News Image Captioning Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis
  2. 2025Multilingual Training-Free Remote Sensing Image Captioning
  3. 2024Brotherhood at WMT 2024: Leveraging LLM-Generated Contextual Conversations for Cross-Lingual Image Captioning
  4. 2024Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval
  5. 2024MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
  6. 2025Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
  7. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  8. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling