mlsanityData to 7 Oct 2026

Multimodal Modeling For Spoken Language Identification

arXiv 2309.10567 ↗First posted 19 Sep 2023cs.CL cs.LG cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Spoken language identification refers to the task of automatically predicting the spoken language in a given utterance. Conventionally, it is modeled as a speech-based language identification task. Prior techniques have been constrained to a single modality; however in the case of video data there is a wealth of other metadata that may be beneficial for this task. In this work, we propose MuSeLI, a Multimodal Spoken Language Identification method, which delves into the use of various metadata sources to enhance language identification. Our study reveals that metadata such as video title, description and geographic location provide substantial information to identify the spoken language of the multimedia recording. We conduct experiments using two diverse public datasets of YouTube videos, and obtain state-of-the-art results on the language identification task. We additionally conduct an ablation study that describes the distinct contribution of each modality for language recognition.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

20 of 35 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  6. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  7. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  8. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  9. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  10. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  11. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  12. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
8 more
  1. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  2. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  3. 2021Towards Building ASR Systems for the Next Billion Userscited by 45
  4. 2022Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASRcited by 13
  5. 2022Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speechcited by 6
  6. 2023Label Aware Speech Representation Learning For Language Identificationcited by 4
  7. 2021Listen, Read, and Identify: Multimodal Singing Language Identification of Musiccited by 2
  8. 2023Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understandingcited by 1

Built on by

Not cited yet.