# Globetrotter: Connecting Languages by Connecting Images

arXiv: https://arxiv.org/abs/2012.04631 · HTML: https://mlsanity.com/paper/2012.04631/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 6 of 6)

- [mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus](https://mlsanity.com/paper/2406.08707/) (2024)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples](https://mlsanity.com/paper/2303.17517/) (2023)
- [MuMUR : Multilingual Multimodal Universal Retrieval](https://mlsanity.com/paper/2208.11553/) (2022)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding](https://mlsanity.com/paper/2608.26925/) (2026)

## Abstract (arXiv)

Machine translation between many languages at once is highly challenging, since training with ground truth requires supervision between all language pairs, which is difficult to obtain. Our key insight is that, while languages may vary drastically, the underlying visual appearance of the world remains consistent. We introduce a method that uses visual observations to bridge the gap between languages, rather than relying on parallel corpora or topological properties of the representations. We train a model that aligns segments of text from different languages if and only if the images associated with them are similar and each image in turn is well-aligned with its textual description. We train our model from scratch on a new dataset of text in over fifty languages with accompanying images. Experiments show that our method outperforms previous work on unsupervised word and sentence translation using retrieval. Code, models and data are available on globetrotter.cs.columbia.edu.