mlsanityData to 14 Oct 2026

Direct Text to Speech Translation System using Acoustic Units

arXiv 2309.07478 ↗First posted 14 Sep 2023cs.CL cs.LG cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper proposes a direct text to speech translation system using discrete acoustic units. This framework employs text in different source languages as input to generate speech in the target language without the need for text transcriptions in this language. Motivated by the success of acoustic units in previous works for direct speech to speech translation systems, we use the same pipeline to extract the acoustic units using a speech encoder combined with a clustering algorithm. Once units are obtained, an encoder-decoder architecture is trained to predict them. Then a vocoder generates speech from units. Our approach for direct text to speech translation was tested on the new CVSS corpus with two different text mBART models employed as initialisation. The systems presented report competitive performance for most of the language pairs evaluated. Besides, results show a remarkable improvement when initialising our proposed architecture with a model pre-trained with more languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 24 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

20 of 24 references.

References · 20

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  8. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  9. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  12. 2021Direct speech-to-speech translation with discrete unitscited by 149
8 more
  1. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  2. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  3. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  4. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  5. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  6. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  7. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  8. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16

Built on by

Not cited yet.