mlsanityData to 7 Oct 2026

Textless Speech-to-Speech Translation With Limited Parallel Data

arXiv 2305.15405 ↗First posted 24 May 2023cs.CL eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

Existing speech-to-speech translation (S2ST) models fall into two camps: they either leverage text as an intermediate step or require hundreds of hours of parallel speech data. Both approaches are incompatible with textless languages or language pairs with limited parallel data. We present PFB, a framework for training textless S2ST models that require just dozens of hours of parallel speech data. We first pretrain a model on large-scale monolingual speech data, finetune it with a small amount of parallel speech data (20-60 hours), and lastly train with an unsupervised backtranslation objective. We train and evaluate our models for English-to-German, German-to-English and Marathi-to-English translation on three different domains (European Parliament, Common Voice, and All India Radio) with single-speaker synthesized speech. Evaluated using the ASR-BLEU metric, our models achieve reasonable performance on all three domains, with some being within 1-2 points of our higher-resourced topline.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

37 of 59 references.

References · 37

  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
25 more
  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  2. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  3. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  4. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  5. 2021Direct speech-to-speech translation with discrete unitscited by 148
  6. 2021Unsupervised Speech Recognitioncited by 147
  7. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  8. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  9. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  10. 2021Survey of Low-Resource Machine Translationcited by 92
  11. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  12. 2021Towards Building ASR Systems for the Next Billion Userscited by 45
  13. 2022Towards End-to-end Unsupervised Speech Recognitioncited by 44
  14. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  15. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  16. 2022Towards Building Text-To-Speech Systems for the Next Billion Userscited by 27
  17. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  18. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  19. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  20. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  21. 2022DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answeringcited by 15
  22. 2022Simple and Effective Unsupervised Speech Synthesiscited by 12
  23. 2022Simple and Effective Unsupervised Speech Translationcited by 12
  24. 2023Improving Cascaded Unsupervised Speech Translation with Denoising Back-translationcited by 5
  25. 2022A Textless Metric for Speech-to-Speech Comparisoncited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  2. 2024Direct Speech-to-Speech Neural Machine Translation: A Surveycited by 5
  3. 2024TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Datacited by 0