# Duplex Diffusion Models Improve Speech-to-Speech Translation

arXiv: https://arxiv.org/abs/2305.12628 · HTML: https://mlsanity.com/paper/2305.12628/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 22 May 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [The Reversible Residual Network: Backpropagation Without Storing Activations](https://mlsanity.com/paper/1707.04585/) (2017)
- [Dual Diffusion Implicit Bridges for Image-to-Image Translation](https://mlsanity.com/paper/2203.08382/) (2022)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [CoVoST 2 and Massively Multilingual Speech-to-Text Translation](https://mlsanity.com/paper/2007.10310/) (2020)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [CVSS Corpus and Massively Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2201.03713/) (2022)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation](https://mlsanity.com/paper/2204.02967/) (2022)
- [Self-Training for End-to-End Speech Translation](https://mlsanity.com/paper/2006.02490/) (2020)
- [Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation](https://mlsanity.com/paper/2104.06457/) (2021)

## Built on by (most cited first, 5 of 5)

- [R3Mem: Bridging Memory Retention and Retrieval via Reversible Compression](https://mlsanity.com/paper/2502.15957/) (2025)
- [Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?](https://mlsanity.com/paper/2406.07289/) (2024)
- [CTC-based Non-autoregressive Textless Speech-to-Speech Translation](https://mlsanity.com/paper/2406.07330/) (2024)
- [Phonology-Guided Speech-to-Speech Translation for African Languages](https://mlsanity.com/paper/2410.23323/) (2024)
- [Seasoning Generative Models for a Generalization Aftertaste](https://mlsanity.com/paper/2603.18817/) (2026)

## Abstract (arXiv)

Speech-to-speech translation is a typical sequence-to-sequence learning task that naturally has two directions. How to effectively leverage bidirectional supervision signals to produce high-fidelity audio for both directions? Existing approaches either train two separate models or a multitask-learned model with low efficiency and inferior performance. In this paper, we propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex Conformer, so that either end can simultaneously input and output a distinct language's speech. Our model enables reversible speech translation by simply flipping the input and output ends. Experiments show that our model achieves the first success of reversible speech translation with significant improvements of ASR-BLEU scores compared with a list of state-of-the-art baselines.