mlsanityData to 14 Oct 2026

Improving Speech-to-Speech Translation Through Unlabeled Text

arXiv 2210.14514 ↗First posted 26 Oct 2022cs.CL cs.SD eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

Direct speech-to-speech translation (S2ST) is among the most challenging problems in the translation paradigm due to the significant scarcity of S2ST data. While effort has been made to increase the data size from unlabeled speech by cascading pretrained speech recognition (ASR), machine translation (MT) and text-to-speech (TTS) models; unlabeled text has remained relatively under-utilized to improve S2ST. We propose an effective way to utilize the massive existing unlabeled text from different languages to create a large amount of S2ST data to improve S2ST performance by applying various acoustic effects to the generated synthetic data. Empirically our method outperforms the state of the art in Spanish-English translation by up to 2 BLEU. Significant gains by the proposed method are demonstrated in extremely low-resource settings for both Spanish-English and Russian-English translations.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

28 of 36 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Layer Normalizationcited by 5,669
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  9. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
16 more
  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  2. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  4. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  5. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  6. 2021Direct speech-to-speech translation with discrete unitscited by 149
  7. 2019Self-Training for End-to-End Speech Recognitioncited by 129
  8. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  9. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  10. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  11. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  12. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  13. 2020Cross-lingual Retrieval for Iterative Self-Supervised Trainingcited by 54
  14. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 44
  15. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  16. 2017Towards speech-to-text translation without speech recognitioncited by 11

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 3.

8 papers.

  1. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  2. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  3. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  4. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  5. 2024UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalizationcited by 8
  6. 2023Multilingual Speech-to-Speech Translation into Multiple Target Languagescited by 5
  7. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  8. 2024TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Datacited by 0