mlsanityData to 14 Oct 2026

Self-training and Pre-training are Complementary for Speech Recognition

arXiv 2010.11430 ↗First posted 22 Oct 2020cs.LG cs.SD eess.AScited by 99

Not on the map. Why?

Abstract (arXiv)

Self-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data. However, it is not clear whether they learn similar patterns or if they can be effectively combined. In this paper, we show that pseudo-labeling and pre-training with wav2vec 2.0 are complementary in a variety of labeled data setups. Using just 10 minutes of labeled data from Libri-light as well as 53k hours of unlabeled data from LibriVox achieves WERs of 3.0%/5.2% on the clean and other test sets of Librispeech - rivaling the best published systems trained on 960 hours of labeled data only a year ago. Training on all labeled data of Librispeech achieves WERs of 1.5%/3.1%.

Key facts. Cited by 99 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 24; 2022: 34; 2023: 21; 2024: 11; 2025: 4; 2026: 5.

99 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2021Going deeper with Image Transformerscited by 764
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  5. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 471
  6. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  7. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  8. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  9. 2021Direct speech-to-speech translation with discrete unitscited by 149
  10. 2021Unsupervised Speech Recognitioncited by 148
  11. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  12. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
38 more
  1. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  2. 2022Self-Training: A Surveycited by 86
  3. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  4. 2021Simple and Effective Zero-shot Cross-lingual Phoneme Recognitioncited by 79
  5. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  6. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  7. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  8. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  9. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  10. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  11. 2022Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 32
  12. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentationcited by 31
  13. 2022Do self-supervised speech models develop human-like perception biases?cited by 28
  14. 2023Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalizationcited by 28
  15. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  16. 2021Pseudo-Labeling for Massively Multilingual Speech Recognitioncited by 24
  17. 2022Sign Language Video Retrieval with Free-Form Textual Queriescited by 23
  18. 2022XTREME-S: Evaluating Cross-lingual Speech Representationscited by 22
  19. 2021Scaling Laws for Acoustic Modelscited by 21
  20. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 21
  21. 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19
  22. 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
  23. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  24. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  25. 2021Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Modelcited by 15
  26. 2023CARLA: Self-supervised Contrastive Representation Learning for Time Series Anomaly Detectioncited by 15
  27. 2024Measuring the Accuracy of Automatic Speech Recognition Solutionscited by 15
  28. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  29. 2023Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challengescited by 14
  30. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  31. 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
  32. 2021Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understandingcited by 12
  33. 2021On the Use of External Data for Spoken Named Entity Recognitioncited by 12
  34. 2022Automatic Speech Recognition Datasets in Cantonese: A Survey and New Datasetcited by 12
  35. 2021Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learningcited by 10
  36. 2022Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessmentcited by 10
  37. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASRcited by 10
  38. 2023Tuning Large language model for End-to-end Speech Translationcited by 9

50 most cited of 99 All 99 (JSON)