mlsanityData to 7 Oct 2026

Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech

arXiv 1804.03052 ↗First posted 9 Apr 2018cs.CL cs.SD eess.AScited by 26

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore the learning of neural network embeddings for natural images and speech waveforms describing the content of those images. These embeddings are learned directly from the waveforms without the use of linguistic transcriptions or conventional speech recognition technology. While prior work has investigated this setting in the monolingual case using English speech data, this work represents the first effort to apply these techniques to languages beyond English. Using spoken captions collected in English and Hindi, we show that the same model architecture can be successfully applied to both languages. Further, we demonstrate that training a multilingual model simultaneously on both languages offers improved performance over the monolingual models. Finally, we show that these models are capable of performing semantic cross-lingual speech-to-speech retrieval.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 4; 2024: 5; 2025: 1; 2026: 2.

26 papers.

Sort:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  3. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  4. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  5. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  6. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  7. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  8. 2024SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answeringcited by 12
  9. 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
  10. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  11. 20223MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videoscited by 8
  12. 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
14 more
  1. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  2. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  3. 2023Video Recognition in Portrait Modecited by 6
  4. 2023Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samplescited by 4
  5. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  6. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  7. 2021Spoken ObjectNet: A Bias-Controlled Spoken Caption Datasetcited by 2
  8. 2024Translating speech with just imagescited by 2
  9. 2023Visually grounded few-shot word acquisition with fewer shotscited by 1
  10. 2025Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Modelscited by 1
  11. 2026Connecting Speech to Words through Imagescited by 1
  12. 2024Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrievalcited by 0
  13. 2024Improved Visually Prompted Keyword Localisation in Real Low-Resource Settingscited by 0
  14. 2026Mapping Written Words to Spoken Words in a Different Language Using Only Visual Groundingcited by 0