mlsanityData to 7 Oct 2026

Back Translation for Speech-to-text Translation Without Transcripts

arXiv 2305.08709 ↗First posted 15 May 2023cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

The success of end-to-end speech-to-text translation (ST) is often achieved by utilizing source transcripts, e.g., by pre-training with automatic speech recognition (ASR) and machine translation (MT) tasks, or by introducing additional ASR and MT data. Unfortunately, transcripts are only sometimes available since numerous unwritten languages exist worldwide. In this paper, we aim to utilize large amounts of target-side monolingual data to enhance ST without transcripts. Motivated by the remarkable success of back translation in MT, we develop a back translation algorithm for ST (BT4ST) to synthesize pseudo ST data from monolingual target data. To ease the challenges posed by short-to-long generation and one-to-many mapping, we introduce self-supervised discrete units and achieve back translation by cascading a target-to-unit model and a unit-to-speech model. With our synthetic ST data, we achieve an average boost of 2.3 BLEU on MuST-C En-De, En-Fr, and En-Es datasets. More experiments show that our method is especially effective in low-resource scenarios.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 55 of its 80 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

55 of 80 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  10. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  12. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
38 more
  1. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  2. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  3. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  4. 2021Direct speech-to-speech translation with discrete unitscited by 148
  5. 2022Generative Spoken Dialogue Language Modelingcited by 125
  6. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  7. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  8. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  9. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  10. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  11. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  12. 2019Tagged Back-Translationcited by 76
  13. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  14. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  15. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  16. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  17. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  18. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  19. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  20. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  21. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  22. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  23. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  24. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  25. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  26. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  27. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  28. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  29. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  30. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  31. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  32. 2019Analyzing ASR pretraining for low-resource speech-to-text translationcited by 27
  33. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  34. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  35. 2020Dynamic Data Selection and Weighting for Iterative Back-Translationcited by 25
  36. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  37. 2021Meta Back-translationcited by 16
  38. 2022Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Unitscited by 15

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 4; 2025: 1.

9 papers.

  1. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  2. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  3. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  4. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  5. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  6. 2024Representation Purification for End-to-End Speech Translationcited by 3
  7. 2025How to Evaluate Speech Translation with Source-Aware Neural MT Metricscited by 3
  8. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
  9. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1