mlsanityData to 7 Oct 2026

Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval

arXiv 2410.02027 ↗First posted 2 Oct 2024cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

There is a scarcity of multilingual vision-language models that properly account for the perceptual differences that are reflected in image captions across languages and cultures. In this work, through a multimodal, multilingual retrieval case study, we quantify the existing lack of model flexibility. We empirically show performance gaps between training on captions that come from native German perception and captions that have been either machine-translated or human-translated from English into German. To address these gaps, we further propose and evaluate caption augmentation strategies. While we achieve mean recall improvements (+1.3), gaps still remain, indicating an open area of future work for the community.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 29 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

15 of 29 references.

References · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  6. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  7. 2023Improving CLIP Training with Language Rewritescited by 216
  8. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  9. 2018Neural Baby Talkcited by 107
  10. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  11. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  12. 2019ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translationcited by 38
3 more
  1. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  2. 2016Stereotyping and Bias in the Flickr30K Datasetcited by 25
  3. 2018Lessons learned in multilingual grounded language learningcited by 8

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2025JEEM: Vision-Language Understanding in Four Arabic Dialectscited by 7
  3. 2025Considering Length Diversity in Retrieval-Augmented Summarizationcited by 2
  4. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modelingcited by 1
  5. 2025GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning
  2. 2024Why do LLaVA Vision-Language Models Reply to Images in English?
  3. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
  4. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation
  5. 2025Multilingual Vision-Language Models, A Survey
  6. 2025CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation
  7. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
  8. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data