mlsanityData to 7 Oct 2026

A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation

arXiv 2502.00374 ↗First posted 1 Feb 2025cs.CL cs.CV cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

Current research in speech-to-speech translation (S2ST) primarily concentrates on translation accuracy and speech naturalness, often overlooking key elements like paralinguistic information, which is essential for conveying emotions and attitudes in communication. To address this, our research introduces a novel, carefully curated multilingual dataset from various movie audio tracks. Each dataset pair is precisely matched for paralinguistic information and duration. We enhance this by integrating multiple prosody transfer techniques, aiming for translations that are accurate, natural-sounding, and rich in paralinguistic details. Our experimental results confirm that our model retains more paralinguistic information from the source speech while maintaining high standards of translation accuracy and naturalness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 1 Feb 2025.

Builds on

23 of 30 references.

References · 23

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  5. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  7. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  8. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  9. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  10. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  11. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  12. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
11 more
  1. 2021Direct speech-to-speech translation with discrete unitscited by 148
  2. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  3. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  4. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  5. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  6. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  7. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  8. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
  9. 2022Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videoscited by 18
  10. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  11. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10

Built on by

Not cited yet.