mlsanityData to 7 Oct 2026

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

arXiv 2405.17809 ↗First posted 28 May 2024cs.CL cs.AI cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

There is a rising interest and trend in research towards directly translating speech from one language to another, known as end-to-end speech-to-speech translation. However, most end-to-end models struggle to outperform cascade models, i.e., a pipeline framework by concatenating speech recognition, machine translation and text-to-speech models. The primary challenges stem from the inherent complexities involved in direct translation tasks and the scarcity of data. In this study, we introduce a novel model framework TransVIP that leverages diverse datasets in a cascade fashion yet facilitates end-to-end inference through joint probability. Furthermore, we propose two separated encoders to preserve the speaker's voice characteristics and isochrony from the source speech during the translation process, making it highly suitable for scenarios such as video dubbing. Our experiments on the French-English language pair demonstrate that our model outperforms the current state-of-the-art speech-to-speech translation model.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

49 of 60 references.

References · 49

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
37 more
  1. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  2. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  3. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  7. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  8. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  9. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  10. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  11. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  12. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  13. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  14. 2021Direct speech-to-speech translation with discrete unitscited by 148
  15. 2022Generative Spoken Dialogue Language Modelingcited by 125
  16. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  17. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  18. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  19. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  20. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  21. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  22. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  23. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  24. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  25. 2021Deep Learning Based Assessment of Synthetic Speech Naturalnesscited by 35
  26. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  27. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  28. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  29. 2022VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbingcited by 20
  30. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  31. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  32. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  33. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
  34. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  35. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  36. 2023Summarization with Precise Length Controlcited by 4
  37. 2023Improving Isochronous Machine Translation with Target Factors and Auxiliary Counterscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 3.

10 papers.

  1. 2025Step-Audio 2 Technical Reportcited by 101
  2. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 6
  3. 2024Direct Speech-to-Speech Neural Machine Translation: A Surveycited by 5
  4. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  5. 2025StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservationcited by 1
  6. 2026STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelitycited by 1
  7. 2024Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languagescited by 0
  8. 2025Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbingcited by 0
  9. 2026Benchmarking Speech-to-Speech Translation Modelscited by 0
  10. 2026DuraS2ST: Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translationcited by 0