mlsanityData to 7 Oct 2026

CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning

arXiv 2406.00021 ↗First posted 23 May 2024cs.CL cs.SD eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

This paper presents CrossVoice, a novel cascade-based Speech-to-Speech Translation (S2ST) system employing advanced ASR, MT, and TTS technologies with cross-lingual prosody preservation through transfer learning. We conducted comprehensive experiments comparing CrossVoice with direct-S2ST systems, showing improved BLEU scores on tasks such as Fisher Es-En, VoxPopuli Fr-En and prosody preservation on benchmark datasets CVSS-T and IndicTTS. With an average mean opinion score of 3.75 out of 4, speech synthesized by CrossVoice closely rivals human speech on the benchmark, highlighting the efficacy of cascade-based systems and transfer learning in multilingual S2ST with prosody transfer.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 14 of its 23 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

14 of 23 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  6. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  7. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  8. 2021Direct speech-to-speech translation with discrete unitscited by 148
  9. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  10. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
  11. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  12. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
2 more
  1. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
  2. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learningcited by 2