mlsanityData to 14 Oct 2026

Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation

arXiv 2406.10223 ↗First posted 14 Jun 2024cs.LG cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce DiffuseST, a low-latency, direct speech-to-speech translation system capable of preserving the input speaker's voice zero-shot while translating from multiple source languages into English. We experiment with the synthesizer component of the architecture, comparing a Tacotron-based synthesizer to a novel diffusion-based synthesizer. We find the diffusion-based synthesizer to improve MOS and PESQ audio quality metrics by 23% each and speaker similarity by 5% while maintaining comparable BLEU scores. Despite having more than double the parameter count, the diffusion synthesizer has lower latency, allowing the entire model to run more than 5× faster than real-time.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 32 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

29 of 32 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2022High Fidelity Neural Audio Compressioncited by 687
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
17 more
  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  3. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  4. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  5. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 152
  6. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  7. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  8. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  9. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  10. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  11. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  12. 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 59
  13. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  14. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  15. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  16. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  17. 2023StyleS2ST: Zero-shot Style Transfer for Direct Speech-to-speech Translationcited by 5

Built on by

Not cited yet.