mlsanityData to 14 Oct 2026

Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models

arXiv 2509.17523 ↗First posted 22 Sep 2025cs.CL eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks such as speech recognition, particularly in monolingual settings. However, multilingual SSL models tend to underperform their monolingual counterparts on each individual language, especially in multilingual scenarios with few languages such as the bilingual setting. In this work, we investigate a novel approach to reduce this performance gap by introducing limited visual grounding into bilingual speech SSL models. Our results show that visual grounding benefits both monolingual and bilingual models, with especially pronounced gains for the latter, reducing the multilingual performance gap on zero-shot phonetic discrimination from 31.5% for audio-only models to 8.04% with grounding.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

19 of 26 references.

References · 19

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  9. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  10. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  11. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  12. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
7 more
  1. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  2. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  3. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  4. 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
  5. 2025fastabx: A library for efficient computation of ABX discriminabilitycited by 7
  6. 2024A Comparative Analysis of Bilingual and Trilingual Wav2Vec Models for Automatic Speech Recognition in Multilingual Oral History Archivescited by 3
  7. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Language Discrimination Improves Linguistic Learning in Multilingual Speech Modelscited by 0