mlsanityData to 13 Oct 2026

Enhancing Speech-to-Speech Translation with Multiple TTS Targets

arXiv 2304.04618 ↗First posted 10 Apr 2023cs.SD cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

It has been known that direct speech-to-speech translation (S2ST) models usually suffer from the data scarcity issue because of the limited existing parallel materials for both source and target speech. Therefore to train a direct S2ST system, previous works usually utilize text-to-speech (TTS) systems to generate samples in the target language by augmenting the data from speech-to-text translation (S2TT). However, there is a limited investigation into how the synthesized target speech would affect the S2ST models. In this work, we analyze the effect of changing synthesized target speech for direct S2ST models. We find that simply combining the target speech from different TTS systems can potentially improve the S2ST performances. Following that, we also propose a multi-task framework that jointly optimizes the S2ST system with multiple targets from different TTS systems. Extensive experiments demonstrate that our proposed framework achieves consistent improvements (2.8 BLEU) over the baselines on the Fisher Spanish-English dataset.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

28 of 38 references.

References · 28

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,793
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,148
  4. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,033
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 803
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 715
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 657
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 559
  9. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 339
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 337
  11. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  12. 2021Direct speech-to-speech translation with discrete unitscited by 148
16 more
  1. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  2. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  3. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  4. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  5. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 70
  6. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  7. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  8. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  9. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  10. 2020StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalizationcited by 26
  11. 2021S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representationscited by 23
  12. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  13. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  14. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  15. 2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9
  16. 2022LibriS2S: A German-English Speech-to-Speech Translation Corpuscited by 1

Built on by

Not cited yet.