mlsanityData to 7 Oct 2026

Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation

arXiv 2203.13339 ↗First posted 24 Mar 2022cs.CL cs.LG cs.SDcited by 20

Not on the map. Why?

Abstract (arXiv)

End-to-end speech-to-speech translation (S2ST) without relying on intermediate text representations is a rapidly emerging frontier of research. Recent works have demonstrated that the performance of such direct S2ST systems is approaching that of conventional cascade S2ST when trained on comparable datasets. However, in practice, the performance of direct S2ST is bounded by the availability of paired S2ST training data. In this work, we explore multiple approaches for leveraging much more widely available unsupervised and weakly-supervised speech and text data to improve the performance of direct S2ST based on Translatotron 2. With our most effective approaches, the average translation quality of direct S2ST on 21 language pairs on the CVSS-C corpus is improved by +13.6 BLEU (or +113% relatively), as compared to the previous state-of-the-art trained without additional data. The improvements on low-resource language are even more significant (+398% relatively on average). Our comparative studies suggest future research directions for S2ST and speech representation learning.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

39 of 53 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  12. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
27 more
  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  2. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  3. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  4. 2021Direct speech-to-speech translation with discrete unitscited by 148
  5. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  6. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  7. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  8. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  9. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  10. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  11. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  12. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  13. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  14. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  15. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  16. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  17. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  18. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  19. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  20. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  21. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  22. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  23. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  24. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  25. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  26. 2020Self-Supervised Representations Improve End-to-End Speech Translationcited by 21
  27. 2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 9; 2024: 2; 2025: 3.

20 papers.

Sort:

By area:

  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  3. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  4. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  5. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  6. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  7. 2025SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Modelscited by 14
  8. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  9. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  10. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  11. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  12. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 6
8 more
  1. 2023Duplex Diffusion Models Improve Speech-to-Speech Translationcited by 5
  2. 2023Towards cross-language prosody transfer for dialogcited by 5
  3. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  4. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  5. 2022Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Featurescited by 1
  6. 2025End-to-End Speech Translation for Low-Resource Languages Using Weakly Labeled Datacited by 1
  7. 2023Enhancing Speech-to-Speech Translation with Multiple TTS Targetscited by 0
  8. 2024Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translationcited by 0