mlsanityData to 14 Oct 2026

CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning

arXiv 2507.20411 ↗First posted 27 Jul 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Multilingual vision-language models have made significant strides in image captioning, yet they still lag behind their English counterparts due to limited multilingual training data and costly large-scale model parameterization. Retrieval-augmented generation (RAG) offers a promising alternative by conditioning caption generation on retrieved examples in the target language, reducing the need for extensive multilingual training. However, multilingual RAG captioning models often depend on retrieved captions translated from English, which can introduce mismatches and linguistic biases relative to the source language. We introduce CONCAP, a multilingual image captioning model that integrates retrieved captions with image-specific concepts, enhancing the contextualization of the input image and grounding the captioning process across different languages. Experiments on the XM3600 dataset indicate that CONCAP enables strong performance on low- and mid-resource languages, with highly reduced data requirements. Our findings highlight the effectiveness of concept-aware retrieval augmentation in bridging multilingual performance gaps.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 38 references that are in the map. First posted on arXiv 27 Jul 2025.

Builds on

27 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024What matters when building vision-language models?3 May
  2. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs13 Jul
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
12 more
  1. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  2. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  3. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  4. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  5. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  6. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  7. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  8. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  9. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  10. 2020fugashi, a Tool for Tokenizing Japanese in Pythoncited by 17
  11. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  12. 2024Understanding Retrieval Robustness for Retrieval-Augmented Image Captioningcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval
  2. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling
  3. 2023Retrieval-augmented Image Captioning
  4. 2024Towards Retrieval-Augmented Architectures for Image Captioning
  5. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG
  6. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
  7. 2026Retrieval-Augmented Long-Context Translation for Cultural Image Captioning: Gators submission for AmericasNLP 2026 shared task
  8. 2025Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models