mlsanityData to 14 Oct 2026

DUB: Discrete Unit Back-translation for Speech Translation

arXiv 2305.11411 ↗First posted 19 May 2023cs.CL cs.SD eess.AScited by 18

Not on the map. Why?

Abstract (arXiv)

How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST. Recently, the approach of representing speech with unsupervised discrete units yields a new way to ease the modality problem. This motivates us to propose Discrete Unit Back-translation (DUB) to answer two questions: (1) Is it better to represent speech with discrete units than with continuous features in direct ST? (2) How much benefit can useful MT techniques bring to ST? With DUB, the back-translation technique can successfully be applied on direct ST and obtains an average boost of 5.5 BLEU on MuST-C En-De/Fr/Es. In the low-resource language scenario, our method achieves comparable performance to existing methods that rely on large-scale external data. Code and models are available at https://github.com/0nutation/DUB.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 55 of its 77 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

55 of 77 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  12. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
38 more
  1. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  2. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  3. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  4. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  6. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  8. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  9. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  10. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  11. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  12. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  13. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  14. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  15. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  16. 2019Tagged Back-Translationcited by 77
  17. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  18. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  19. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  20. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  21. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  22. 2019Generalized Data Augmentation for Low-Resource Translationcited by 53
  23. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  24. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  25. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  26. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  27. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 46
  28. 2017The Zero Resource Speech Challenge 2017cited by 45
  29. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  30. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  31. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  32. 2020Self-Training for End-to-End Speech Translationcited by 36
  33. 2018Back-Translation-Style Data Augmentation for End-to-End ASRcited by 28
  34. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  35. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  36. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  37. 2020NeurST: Neural Speech Translation Toolkitcited by 23
  38. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 8; 2025: 3.

18 papers.

Sort:

By area:

  1. 2023SeqXGPT: Sentence-Level AI-Generated Text Detectioncited by 56
  2. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 47
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  4. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  5. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  6. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  7. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  8. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 17
  9. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  10. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  11. 2023ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understandingcited by 6
  12. 2025Meaningful Pose-Based Sign Language Evaluationcited by 6
6 more
  1. 2024BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generationcited by 4
  2. 2023Textless Speech-to-Speech Translation With Limited Parallel Datacited by 3
  3. 2024A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Modelscited by 3
  4. 2024Representation Purification for End-to-End Speech Translationcited by 3
  5. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2
  6. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1