mlsanityData to 14 Oct 2026

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

arXiv 2109.13226 ↗First posted 27 Sep 2021eess.AS cs.CL cs.LGcited by 88

Not on the map. Why?

Abstract (arXiv)

We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 64 of its 106 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

64 of 106 references.

References · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
38 more
  1. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  2. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  4. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  5. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  6. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
  7. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  8. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  9. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  10. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  11. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  12. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  13. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 132
  14. 2019Self-Training for End-to-End Speech Recognitioncited by 129
  15. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  16. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  17. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  18. 2020Unsupervised pretraining transfers well across languagescited by 101
  19. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  20. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  21. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  22. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 83
  23. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  24. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 73
  25. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  26. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 69
  27. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  28. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  29. 2019SpecAugment on Large Scale Datasetscited by 60
  30. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  31. 2017Unsupervised Learning of Semantic Audio Representationscited by 44
  32. 2020Learning Robust and Multilingual Speech Representationscited by 39
  33. 2021Multi-Format Contrastive Learning of Audio Representationscited by 39
  34. 2018Toward domain-invariant speech recognition via large scale trainingcited by 35
  35. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  36. 2020Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Trainingcited by 30
  37. 2021On the limit of English conversational speech recognitioncited by 30
  38. 2019Lessons from Building Acoustic Models with a Million Hours of Speechcited by 27

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 24; 2023: 31; 2024: 16; 2025: 5; 2026: 3.

88 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  5. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  6. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  7. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  8. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  9. 2023End-to-End Speech Recognition: A Surveycited by 129
  10. 2023Joint Audio and Speech Understandingcited by 100
  11. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 90
  12. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
38 more
  1. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  2. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  3. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  4. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 45
  5. 2023Transformers in Speech Processing: A Surveycited by 39
  6. 2024CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptionscited by 34
  7. 2021Universal Paralinguistic Speech Representations Using Self-Supervised Conformerscited by 32
  8. 2024Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Modelscited by 29
  9. 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
  10. 2023BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for Short-Term Load Forecastingcited by 26
  11. 2022Pseudo Label Is Better Than Human Labelcited by 25
  12. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  13. 2023From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognitioncited by 23
  14. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18
  15. 2023Self-supervised representations in speech-based depression detectioncited by 18
  16. 2022TRILLsson: Distilled Universal Paralinguistic Speech Representationscited by 16
  17. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16
  18. 2024HeAR -- Health Acoustic Representationscited by 15
  19. 2023ADCNet: Learning from Raw Radar Data via Distillationcited by 14
  20. 2024Anatomy of Industrial Scale Multilingual ASRcited by 14
  21. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  22. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  23. 2024Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Studycited by 12
  24. 2021Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognitioncited by 10
  25. 2022ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversioncited by 10
  26. 2022Improving Deliberation by Text-Only and Semi-Supervised Trainingcited by 10
  27. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASRcited by 10
  28. 2023Massively Multilingual Shallow Fusion with Large Language Modelscited by 10
  29. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  30. 2022Production federated keyword spotting via distillation, filtering, and joint federated-centralized trainingcited by 9
  31. 2021Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASRcited by 7
  32. 2022VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognitioncited by 7
  33. 2023Modular Domain Adaptation for Conformer-Based Streaming ASRcited by 7
  34. 2022Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learningcited by 6
  35. 2024A Survey of RWKVcited by 6
  36. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 5
  37. 2021Multilingual Speech Recognition using Knowledge Transfer across Learning Processescited by 4
  38. 2023Pre-Finetuning for Few-Shot Emotional Speech Recognitioncited by 4

50 most cited of 88 All 88 (JSON)