mlsanityData to 7 Oct 2026

Multilingual Speech-to-Speech Translation into Multiple Target Languages

arXiv 2307.08655 ↗First posted 17 Jul 2023cs.CL cs.SD eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Speech-to-speech translation (S2ST) enables spoken communication between people talking in different languages. Despite a few studies on multilingual S2ST, their focus is the multilinguality on the source side, i.e., the translation from multiple source languages to one target language. We present the first work on multilingual S2ST supporting multiple target languages. Leveraging recent advance in direct S2ST with speech-to-unit and vocoder, we equip these key components with multilingual capability. Speech-to-masked-unit (S2MU) is the multilingual extension of S2U, which applies masking to units which don't belong to the given target language to reduce the language interference. We also propose multilingual vocoder which is trained with language embedding and the auxiliary loss of language identification. On benchmark translation testsets, our proposed multilingual model shows superior performance than bilingual models in the translation from English into 16 target languages.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 20 of its 22 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

20 of 22 references.

References · 20

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  5. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  6. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  7. 2019Massively Multilingual Neural Machine Translationcited by 246
  8. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  9. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  10. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  11. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  12. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
8 more
  1. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  2. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  3. 2019CSS10: A Collection of Single Speaker Speech Datasets for 10 Languagescited by 54
  4. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  5. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  6. 2020One Model, Many Languages: Meta-learning for Multilingual Text-to-Speechcited by 28
  7. 2022Improving Speech-to-Speech Translation Through Unlabeled Textcited by 8
  8. 2021FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Taskcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 0; 2026: 1.

5 papers.

  1. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  2. 2024SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thoughtcited by 9
  3. 2024Direct Speech-to-Speech Neural Machine Translation: A Surveycited by 5
  4. 2024Direct Punjabi to English speech translation using discrete unitscited by 1
  5. 2026Multilingual Multi-Speaker Unit Vocoders: A Systematic Analysis of Discrete Speech Representationscited by 0