mlsanityData to 7 Oct 2026

Large-Scale Self- and Semi-Supervised Learning for Speech Translation

arXiv 2104.06678 ↗First posted 14 Apr 2021cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

In this paper, we improve speech translation (ST) through effectively leveraging large quantities of unlabeled speech and text data in different and complementary ways. We explore both pretraining and self-training by using the large Libri-Light speech audio corpus and language modeling with CommonCrawl. Our experiments improve over the previous state of the art by 2.6 BLEU on average on all four considered CoVoST 2 language pairs via a simple recipe of combining wav2vec 2.0 pretraining, a single iteration of self-training and decoding with a language model. Different to existing work, our approach does not leverage any other supervision than ST data. Code and models will be publicly released.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 40 of its 47 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

40 of 47 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
28 more
  1. 2015Attention-Based Models for Speech Recognitioncited by 400
  2. 2020Rethinking Pre-training and Self-trainingcited by 352
  3. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  4. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  5. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  6. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  7. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  8. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  9. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  10. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  11. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  12. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  13. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  14. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
  15. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  16. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  17. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  18. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  19. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  20. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  21. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  22. 2019Multilingual End-to-End Speech Translationcited by 37
  23. 2020Self-Training for End-to-End Speech Translationcited by 36
  24. 2019Analyzing ASR pretraining for low-resource speech-to-text translationcited by 27
  25. 2020Self-Supervised Representations Improve End-to-End Speech Translationcited by 21
  26. 2019One-To-Many Multilingual End-to-end Speech Translationcited by 17
  27. 2019Exploring Phoneme-Level Speech Representations for End-to-End Speech Translationcited by 15
  28. 2020SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 13; 2023: 10; 2024: 3; 2025: 2; 2026: 2.

36 papers.

Sort:

By area:

  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  2. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  3. 2021Unsupervised Speech Recognitioncited by 147
  4. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  5. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  6. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  7. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  8. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  9. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  10. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  11. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  12. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
24 more
  1. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  2. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  3. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  4. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  5. 2023SlothSpeech: Denial-of-service Attack Against Speech Recognition Modelscited by 9
  6. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  7. 2022Improving Speech-to-Speech Translation Through Unlabeled Textcited by 8
  8. 2022Mu2SLAM: Multitask, Multilingual Speech and Language Modelscited by 7
  9. 2024Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Schedulingcited by 7
  10. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  11. 2023Improving Cascaded Unsupervised Speech Translation with Denoising Back-translationcited by 5
  12. 2023Benchmarking Diverse-Modal Entity Linking with Generative Modelscited by 5
  13. 2024CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleavingcited by 4
  14. 2025When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translationcited by 4
  15. 2022Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Datacited by 3
  16. 2022Hearing voices at the National Library -- a speech corpus and acoustic model for the Swedish languagecited by 2
  17. 2022An End-to-End OCR Framework for Robust Arabic-Handwriting Recognition using a Novel Transformers-based Model and an Innovative 270 Million-Words Multi-Font Corpus of Classical Arabic with Diacriticscited by 2
  18. 2023Fast FixMatch: Faster Semi-Supervised Learning with Curriculum Batch Sizecited by 2
  19. 2023An Investigation of Representation and Allocation Harms in Contrastive Learningcited by 2
  20. 2023Form follows Function: Text-to-Text Conditional Graph Generation based on Functional Requirementscited by 1
  21. 2026How to Leverage Synthetic Speech for LLM-Based ASR Systems?cited by 1
  22. 2024A Comparative Study of Pre-training and Self-trainingcited by 0
  23. 2025LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Datacited by 0
  24. 2026When Synthetic Speech Is All You Have: Better Call GRPOcited by 0