mlsanityData to 14 Oct 2026

Duplex Diffusion Models Improve Speech-to-Speech Translation

arXiv 2305.12628 ↗First posted 22 May 2023cs.CL cs.LG cs.SDcited by 5

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Speech-to-speech translation is a typical sequence-to-sequence learning task that naturally has two directions. How to effectively leverage bidirectional supervision signals to produce high-fidelity audio for both directions? Existing approaches either train two separate models or a multitask-learned model with low efficiency and inferior performance. In this paper, we propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex Conformer, so that either end can simultaneously input and output a distinct language's speech. Our model enables reversible speech translation by simply flipping the input and output ends. Experiments show that our model achieves the first success of reversible speech translation with significant improvements of ASR-BLEU scores compared with a list of state-of-the-art baselines.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 32 of its 48 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

32 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2016Layer Normalizationcited by 5,669
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  10. 2018Self-Attention with Relative Position Representationscited by 1,120
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  12. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
19 more
  1. 2020On Layer Normalization in the Transformer Architecturecited by 922
  2. 2016Language Modeling with Gated Convolutional Networkscited by 702
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  4. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  5. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 217
  6. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 213
  7. 2021Direct speech-to-speech translation with discrete unitscited by 149
  8. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  9. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  10. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  11. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  12. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  13. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  14. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  15. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 44
  16. 2020Self-Training for End-to-End Speech Translationcited by 36
  17. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 28
  18. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  19. 2021Duplex Sequence-to-Sequence Learning for Reversible Machine Translationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
  2. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  3. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1
  4. 2024Phonology-Guided Speech-to-Speech Translation for African Languagescited by 0
  5. 2026Seasoning Generative Models for a Generalization Aftertastecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
  2. 2023DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation
  3. 2025Posterior Transition Modeling for Unsupervised Diffusion-Based Speech Enhancement
  4. 2026Beyond Two-stage Diffusion TTS: Joint Structure and Content Refinement via Jump Diffusion
  5. 2022BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis
  6. 2024Sample-Efficient Diffusion for Text-To-Speech Synthesis
  7. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  8. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding