mlsanityData to 7 Oct 2026

Generating Synthetic Speech from SpokenVocab for Speech Translation

arXiv 2210.08174 ↗First posted 15 Oct 2022cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Training end-to-end speech translation (ST) systems requires sufficiently large-scale data, which is unavailable for most language pairs and domains. One practical solution to the data scarcity issue is to convert machine translation data (MT) to ST data via text-to-speech (TTS) systems. Yet, using TTS systems can be tedious and slow, as the conversion needs to be done for each MT dataset. In this work, we propose a simple, scalable and effective data augmentation technique, i.e., SpokenVocab, to convert MT data to ST data on-the-fly. The idea is to retrieve and stitch audio snippets from a SpokenVocab bank according to words in an MT sequence. Our experiments on multiple language pairs from Must-C show that this method outperforms strong baselines by an average of 1.83 BLEU scores, and it performs equally well as TTS-generated speech. We also showcase how SpokenVocab can be applied in code-switching ST for which often no TTS systems exit. Our code is available at https://github.com/mingzi151/SpokenVocab

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 15 Oct 2022.

Builds on

19 of 26 references.

References · 19

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  4. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  5. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  6. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  7. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  8. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  9. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  10. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  11. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  12. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
7 more
  1. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  2. 2022End-to-End Speech Translation for Code Switched Speechcited by 26
  3. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  4. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  5. 2022Code-Switching without Switching: Language Agnostic End-to-End Speech Translationcited by 13
  6. 2021Audio declipping performance enhancement via crossfadingcited by 2
  7. 2022Prabhupadavani: A Code-mixed Speech Translation Data for 25 Languagescited by 2

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  3. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  4. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  5. 2024Honest AI: Fine-Tuning "Small" Language Models to Say "I Don't Know", and Reducing Hallucination in RAGcited by 6