mlsanityData to 7 Oct 2026

Pre-training on high-resource speech recognition improves low-resource speech-to-text translation

arXiv 1809.01431 ↗First posted 5 Sep 2018cs.CLcited by 70

Not on the map. Why?

Abstract (arXiv)

We present a simple approach to improve direct speech-to-text translation (ST) when the source language is low-resource: we pre-train the model on a high-resource automatic speech recognition (ASR) task, and then fine-tune its parameters for ST. We demonstrate that our approach is effective by pre-training on 300 hours of English ASR data to improve Spanish-English ST from 10.8 to 20.2 BLEU when only 20 hours of Spanish-English ST training data are available. Through an ablation study, we find that the pre-trained encoder (acoustic model) accounts for most of the improvement, despite the fact that the shared language in these tasks is the target language text, not the source language audio. Applying this insight, we show that pre-training on ASR helps ST even when the ASR language differs from both source and target ST languages: pre-training on French ASR also improves Spanish-English ST. Finally, we show that the approach improves performance on a true low-resource task: pre-training on a combination of English ASR and French ASR improves Mboshi-French ST, where only 4 hours of data are available, from 3.5 to 7.1 BLEU.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 18; 2023: 15; 2024: 7; 2025: 4; 2026: 2.

70 papers.

Sort:

By area:

  1. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  2. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  3. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  4. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  5. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  6. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  7. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  8. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  9. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  10. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
  11. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  12. 2021End-to-End Speech Recognition from Federated Acoustic Modelscited by 30
38 more
  1. 2021Learning When to Translate for Streaming Speechcited by 30
  2. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  3. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  4. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  5. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  6. 2023From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognitioncited by 23
  7. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  8. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  9. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  10. 2024Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learningcited by 20
  11. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  12. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  13. 2021Regularizing End-to-End Speech Translation with Triangular Decomposition Agreementcited by 16
  14. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  15. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  16. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  17. 2022Non-Parametric Domain Adaptation for End-to-End Speech Translationcited by 12
  18. 2022Simple and Effective Unsupervised Speech Translationcited by 12
  19. 2021Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understandingcited by 11
  20. 2021Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translationcited by 11
  21. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  22. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  23. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  24. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  25. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  26. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  27. 2022When Is TTS Augmentation Through a Pivot Language Useful?cited by 7
  28. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  29. 2021Dealing with training and test segmentation mismatch: FBK@IWSLT2021cited by 6
  30. 2021Decision Attentive Regularization to Improve Simultaneous Speech Translation Systemscited by 6
  31. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  32. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  33. 2021Zero-shot Speech Translationcited by 5
  34. 2022FedNST: Federated Noisy Student Training for Automatic Speech Recognitioncited by 5
  35. 2022Generating Synthetic Speech from SpokenVocab for Speech Translationcited by 5
  36. 2021K-Wav2vec 2.0: Automatic Speech Recognition based on Joint Decoding of Graphemes and Syllablescited by 4
  37. 2022Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniquescited by 4
  38. 2022Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generationcited by 4

50 most cited of 70 All 70 (JSON)