# Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech

arXiv: https://arxiv.org/abs/1804.03052 · HTML: https://mlsanity.com/paper/1804.03052/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2018.

## Built on by (most cited first, 26 of 26)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model](https://mlsanity.com/paper/2210.00705/) (2022)
- [Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling](https://mlsanity.com/paper/2202.03543/) (2022)
- [Fast-Slow Transformer for Visually Grounding Speech](https://mlsanity.com/paper/2109.08186/) (2021)
- [Attention-Based Keyword Localisation in Speech using Visual Grounding](https://mlsanity.com/paper/2106.08859/) (2021)
- [SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answering](https://mlsanity.com/paper/2401.13463/) (2024)
- [M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval](https://mlsanity.com/paper/2211.01180/) (2022)
- [Cascaded Multilingual Audio-Visual Learning from Videos](https://mlsanity.com/paper/2111.04823/) (2021)
- [3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos](https://mlsanity.com/paper/2203.14456/) (2022)
- [YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual grounding](https://mlsanity.com/paper/2210.04600/) (2022)
- [Keyword localisation in untranscribed speech using visually grounded speech models](https://mlsanity.com/paper/2202.01107/) (2022)
- [Towards visually prompted keyword localisation for zero-resource spoken languages](https://mlsanity.com/paper/2210.06229/) (2022)
- [Video Recognition in Portrait Mode](https://mlsanity.com/paper/2312.13746/) (2023)
- [Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples](https://mlsanity.com/paper/2303.17517/) (2023)
- [Visually grounded few-shot word learning in low-resource settings](https://mlsanity.com/paper/2306.11371/) (2023)
- [Visually Grounded Speech Models have a Mutual Exclusivity Bias](https://mlsanity.com/paper/2403.13922/) (2024)
- [Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset](https://mlsanity.com/paper/2110.07575/) (2021)
- [Translating speech with just images](https://mlsanity.com/paper/2406.07133/) (2024)
- [Visually grounded few-shot word acquisition with fewer shots](https://mlsanity.com/paper/2305.15937/) (2023)
- [Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models](https://mlsanity.com/paper/2509.17523/) (2025)
- [Connecting Speech to Words through Images](https://mlsanity.com/paper/2606.16807/) (2026)
- [Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrieval](https://mlsanity.com/paper/2408.13705/) (2024)
- [Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings](https://mlsanity.com/paper/2409.06013/) (2024)
- [Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding](https://mlsanity.com/paper/2608.26925/) (2026)

## Abstract (arXiv)

In this paper, we explore the learning of neural network embeddings for natural images and speech waveforms describing the content of those images. These embeddings are learned directly from the waveforms without the use of linguistic transcriptions or conventional speech recognition technology. While prior work has investigated this setting in the monolingual case using English speech data, this work represents the first effort to apply these techniques to languages beyond English. Using spoken captions collected in English and Hindi, we show that the same model architecture can be successfully applied to both languages. Further, we demonstrate that training a multilingual model simultaneously on both languages offers improved performance over the monolingual models. Finally, we show that these models are capable of performing semantic cross-lingual speech-to-speech retrieval.