mlsanityData to 7 Oct 2026

Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition

arXiv 2010.10504 ↗First posted 20 Oct 2020eess.AS cs.LG cs.SDcited by 185

Not on the map. Why?

Abstract (arXiv)

We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 56; 2022: 53; 2023: 40; 2024: 27; 2025: 5; 2026: 4.

185 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021Going deeper with Image Transformerscited by 759
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  6. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  7. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  8. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  9. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  10. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  11. 2021Unsupervised Speech Recognitioncited by 147
  12. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
38 more
  1. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 116
  2. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  3. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  4. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  5. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  6. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  7. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  8. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  9. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  10. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  11. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 66
  12. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  13. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  14. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 60
  15. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  16. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  17. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  18. 2022Towards End-to-end Unsupervised Speech Recognitioncited by 44
  19. 2023Benchmarking and Improving Generator-Validator Consistency of Language Modelscited by 43
  20. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  21. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  22. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  23. 2023Transformers in Speech Processing: A Surveycited by 38
  24. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  25. 2024GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinementcited by 34
  26. 2021Universal Paralinguistic Speech Representations Using Self-Supervised Conformerscited by 32
  27. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  28. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  29. 2022Pseudo Label Is Better Than Human Labelcited by 25
  30. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  31. 2021Semi-Supervised Vision Transformerscited by 24
  32. 2021Pseudo-Labeling for Massively Multilingual Speech Recognitioncited by 23
  33. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  34. 2023Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representationscited by 22
  35. 2022Sign Language Video Retrieval with Free-Form Textual Queriescited by 21
  36. 2021Improved Language Identification Through Cross-Lingual Self-Supervised Learningcited by 20
  37. 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19
  38. 2022Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizerscited by 19

50 most cited of 185 All 185 (JSON)