# DUB: Discrete Unit Back-translation for Speech Translation

arXiv: https://arxiv.org/abs/2305.11411 · HTML: https://mlsanity.com/paper/2305.11411/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 77 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [ESPnet-ST: All-in-One Speech Translation Toolkit](https://mlsanity.com/paper/2004.10234/) (2020)
- [Tagged Back-Translation](https://mlsanity.com/paper/1906.06442/) (2019)
- [Cross-modal Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2205.02444/) (2022)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)

## Built on by (most cited first, 18 of 18)

- [SeqXGPT: Sentence-Level AI-Generated Text Detection](https://mlsanity.com/paper/2310.08903/) (2023)
- [SpeechAlign: Aligning Speech Generation to Human Preferences](https://mlsanity.com/paper/2404.05600/) (2024)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study](https://mlsanity.com/paper/2309.15800/) (2023)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [How Should We Extract Discrete Audio Tokens from Self-Supervised Models?](https://mlsanity.com/paper/2406.10735/) (2024)
- [PolyVoice: Language Models for Speech to Speech Translation](https://mlsanity.com/paper/2306.02982/) (2023)
- [IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities](https://mlsanity.com/paper/2410.08035/) (2024)
- [Translatotron 3: Speech to Speech Translation with Monolingual Data](https://mlsanity.com/paper/2305.17547/) (2023)
- [Tuning Large language model for End-to-end Speech Translation](https://mlsanity.com/paper/2310.02050/) (2023)
- [ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2311.11375/) (2023)
- [Meaningful Pose-Based Sign Language Evaluation](https://mlsanity.com/paper/2510.07453/) (2025)
- [BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation](https://mlsanity.com/paper/2410.14971/) (2024)
- [Textless Speech-to-Speech Translation With Limited Parallel Data](https://mlsanity.com/paper/2305.15405/) (2023)
- [A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Models](https://mlsanity.com/paper/2411.08742/) (2024)
- [Representation Purification for End-to-End Speech Translation](https://mlsanity.com/paper/2412.04266/) (2024)
- [DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignment](https://mlsanity.com/paper/2509.18987/) (2025)
- [Compact Speech Translation Models via Discrete Speech Units Pretraining](https://mlsanity.com/paper/2402.19333/) (2024)

## Abstract (arXiv)

How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST. Recently, the approach of representing speech with unsupervised discrete units yields a new way to ease the modality problem. This motivates us to propose Discrete Unit Back-translation (DUB) to answer two questions: (1) Is it better to represent speech with discrete units than with continuous features in direct ST? (2) How much benefit can useful MT techniques bring to ST? With DUB, the back-translation technique can successfully be applied on direct ST and obtains an average boost of 5.5 BLEU on MuST-C En-De/Fr/Es. In the low-resource language scenario, our method achieves comparable performance to existing methods that rely on large-scale external data. Code and models are available at https://github.com/0nutation/DUB.