mlsanityData to 14 Oct 2026

TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data

arXiv 2401.12992 ↗First posted 17 Jan 2024cs.CL cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Although there has been significant advancement in the field of speech-to-speech translation, conventional models still require language-parallel speech data between the source and target languages for training. In this paper, we introduce TranSentence, a novel speech-to-speech translation without language-parallel speech data. To achieve this, we first adopt a language-agnostic sentence-level speech encoding that captures the semantic information of speech, irrespective of language. We then train our model to generate speech based on the encoded embedding obtained from a language-agnostic sentence-level speech encoder that is pre-trained with various languages. With this method, despite training exclusively on the target language's monolingual data, we can generate target language speech in the inference stage using language-agnostic speech embedding from the source language speech. Furthermore, we extend TranSentence to multilingual speech-to-speech translation. The experimental results demonstrate that TranSentence is superior to other models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 25 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

22 of 25 references.

References · 22

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  6. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  7. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  8. 2021Direct speech-to-speech translation with discrete unitscited by 149
  9. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  10. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  11. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  12. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
10 more
  1. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  2. 2019Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networkscited by 49
  3. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  4. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  5. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  6. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  7. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 11
  8. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  9. 2022Improving Speech-to-Speech Translation Through Unlabeled Textcited by 8
  10. 2023Textless Speech-to-Speech Translation With Limited Parallel Datacited by 3

Built on by

Not cited yet.