# Lessons learned in multilingual grounded language learning

arXiv: https://arxiv.org/abs/1809.07615 · HTML: https://mlsanity.com/paper/1809.07615/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2018.

## Built on by (most cited first, 8 of 8)

- [Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval](https://mlsanity.com/paper/2103.11920/) (2021)
- [Multilingual Multimodal Learning with Machine Translated Text](https://mlsanity.com/paper/2210.13134/) (2022)
- [JEEM: Vision-Language Understanding in Four Arabic Dialects](https://mlsanity.com/paper/2503.21910/) (2025)
- [Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval](https://mlsanity.com/paper/2410.02027/) (2024)
- [Towards Zero-shot Cross-lingual Image Retrieval and Tagging](https://mlsanity.com/paper/2109.07622/) (2021)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling](https://mlsanity.com/paper/2504.14359/) (2025)
- [Multilingual Vision-Language Models, A Survey](https://mlsanity.com/paper/2509.22123/) (2025)

## Abstract (arXiv)

Recent work has shown how to learn better visual-semantic embeddings by leveraging image descriptions in more than one language. Here, we investigate in detail which conditions affect the performance of this type of grounded language learning model. We show that multilingual training improves over bilingual training, and that low-resource languages benefit from training with higher-resource languages. We demonstrate that a multilingual model can be trained equally well on either translations or comparable sentence pairs, and that annotating the same set of images in multiple language enables further improvements via an additional caption-caption ranking objective.