mlsanityData to 7 Oct 2026

Analyzing Speech Unit Selection for Textless Speech-to-Speech Translation

arXiv 2407.18332 ↗First posted 8 Jul 2024eess.AS cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advancements in textless speech-to-speech translation systems have been driven by the adoption of self-supervised learning techniques. Although most state-of-the-art systems adopt a similar architecture to transform source language speech into sequences of discrete representations in the target language, the criteria for selecting these target speech units remains an open question. This work explores the selection process through a study of downstream tasks such as automatic speech recognition, speech synthesis, speaker recognition, and emotion recognition. Interestingly, our findings reveal a discrepancy in the optimization of discrete speech units: units that perform well in resynthesis performance do not necessarily correlate with those that enhance translation efficacy. This discrepancy underscores the nuanced complexity of target feature selection and its impact on the overall performance of speech-to-speech translation systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

20 of 25 references.

References · 20

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  9. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  10. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  11. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  12. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
8 more
  1. 2021Direct speech-to-speech translation with discrete unitscited by 148
  2. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 113
  3. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  4. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  5. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  6. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  7. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  8. 2023Enhancing expressivity transfer in textless speech-to-speech translationcited by 2

Built on by

Not cited yet.