mlsanityData to 7 Oct 2026

SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation

arXiv 2205.08180 ↗First posted 17 May 2022cs.CL cs.LG cs.SDcited by 32

Not on the map. Why?

Abstract (arXiv)

We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10-20ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a sentence (5-10s) such that the embedding vector space is semantically aligned across different languages. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual speech-text and speech-speech associations emerge in its learned representation space. To substantiate our claims, we use SAMU-XLSR speech encoder in combination with a pre-trained LaBSE text sentence encoder for cross-lingual speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual speech-to-speech translation retrieval. We highlight these applications by performing several cross-lingual text and speech translation retrieval tasks across several datasets.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 17 May 2022.

Builds on

38 of 47 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
26 more
  1. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  2. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  3. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  4. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  5. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  6. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  7. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  8. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  9. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  10. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  11. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  12. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  13. 2020Unsupervised pretraining transfers well across languagescited by 100
  14. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  15. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  16. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  17. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  18. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  19. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  20. 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
  21. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  22. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  23. 2020Self-attention encoding and pooling for speaker recognitioncited by 33
  24. 2018Filtering and Mining Parallel Data in a Joint Multilingual Spacecited by 32
  25. 2020A Convolutional Deep Markov Model for Unsupervised Speech Representation Learningcited by 18
  26. 2020The 2020 ESPnet update: new features, broadened applications, performance improvements, and future planscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 9; 2024: 8; 2025: 3; 2026: 6.

32 papers.

Sort:

  1. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  2. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  3. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  4. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  5. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  6. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  7. 2023The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any languagecited by 14
  8. 2024Cross-Lingual Transfer Learning for Speech Translationcited by 11
  9. 2023NAVER LABS Europe's Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Trackcited by 9
  10. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8
  11. 2022On the Use of Semantically-Aligned Speech Representations for Spoken Language Understandingcited by 7
  12. 2024Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialectcited by 7
20 more
  1. 2025SENSE models: an open source solution for multilingual and multimodal semantic-based taskscited by 6
  2. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systemscited by 5
  3. 2022token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Textcited by 3
  4. 2022Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingscited by 2
  5. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translationcited by 2
  6. 2023Semantic enrichment towards efficient speech representationscited by 2
  7. 2023Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfercited by 2
  8. 2024New Semantic Task for the French Spoken Language Understanding MEDIA Benchmarkcited by 2
  9. 2024CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrievalcited by 2
  10. 2025Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documentscited by 2
  11. 2024A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understandingcited by 1
  12. 2025Factorized RVQ-GAN For Disentangled Speech Tokenizationcited by 1
  13. 2023Direct Text to Speech Translation System using Acoustic Unitscited by 0
  14. 2024Analyzing Speech Unit Selection for Textless Speech-to-Speech Translationcited by 0
  15. 2026Towards Comprehensive Semantic Speech Embeddings for Chinese Dialectscited by 0
  16. 2026Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learningcited by 0
  17. 2026Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encodercited by 0
  18. 2026Forewarned is Forearmed: When Non-Sequential Embedding Turns Into an Anomaly Detectorcited by 0
  19. 2026Low resource cross-modal alignment using HGNN to enhance speech representationcited by 0
  20. 2026Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yembacited by 0