mlsanityData to 14 Oct 2026

M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval

arXiv 2211.01180 ↗First posted 2 Nov 2022cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

This work investigates the use of large-scale, English-only pre-trained models (CLIP and HuBERT) for multilingual image-speech retrieval. For non-English image-speech retrieval, we outperform the current state-of-the-art performance by a wide margin both when training separate models for each language, and with a single model which processes speech in all three languages. We identify key differences in model behavior and performance between English and non-English settings, attributable to the English-only pre-training of CLIP and HuBERT, and investigate how fine-tuning the pre-trained models impacts these differences. Finally, we show that our models can be used for mono- and cross-lingual speech-text retrieval and cross-lingual speech-speech retrieval, despite never having seen any parallel speech-text or speech-speech data during training.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

20 of 29 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  8. 2015YFCC100M: The New Data in Multimedia Researchcited by 235
  9. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  10. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 51
  11. 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
  12. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
8 more
  1. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  2. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  3. 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
  4. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  5. 2019Models of Visually Grounded Speech Signal Pay Attention To Nouns: a Bilingual Experiment on English and Japanesecited by 13
  6. 2019On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrievalcited by 8
  7. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  8. 2021Towards Zero-shot Cross-lingual Image Retrieval and Taggingcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5; 2025: 2.

9 papers.

  1. 2024Interface Design for Self-Supervised Speech Modelscited by 5
  2. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  3. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  4. 2024SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image datacited by 2
  5. 2024Translating speech with just imagescited by 2
  6. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  7. 2025Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Modelscited by 1
  8. 2023SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic Spacescited by 0
  9. 2025Unifying Model and Layer Fusion for Speech Foundation Modelscited by 0