mlsanityData to 14 Oct 2026

Pseudo Label Is Better Than Human Label

arXiv 2203.12668 ↗First posted 22 Mar 2022cs.LG cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

State-of-the-art automatic speech recognition (ASR) systems are trained with tens of thousands of hours of labeled speech data. Human transcription is expensive and time consuming. Factors such as the quality and consistency of the transcription can greatly affect the performance of the ASR models trained with these data. In this paper, we show that we can train a strong teacher model to produce high quality pseudo labels by utilizing recent self-supervised and semi-supervised learning techniques. Specifically, we use JUST (Joint Unsupervised/Supervised Training) and iterative noisy student teacher training to train a 600 million parameter bi-directional teacher model. This model achieved 4.0% word error rate (WER) on a voice search task, 11.1% relatively better than a baseline. We further show that by using this strong teacher model to generate high-quality pseudo labels for training, we can achieve 13.6% relative WER reduction (5.9% to 5.1%) for a streaming model compared to using human labels.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

26 of 38 references.

References · 26

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  6. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  10. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  11. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  12. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
14 more
  1. 2020Hybrid Autoregressive Transducer (hat)cited by 108
  2. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 100
  3. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  4. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  5. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  6. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 69
  7. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  8. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 53
  9. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  10. 2018Toward domain-invariant speech recognition via large scale trainingcited by 35
  11. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  12. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  13. 2021Learning Word-Level Confidence For Subword End-to-End ASRcited by 22
  14. 2020Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Datacited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 6; 2025: 5; 2026: 3.

25 papers.

Sort:

By area:

  1. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  2. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  3. 2025ZIPA: A family of efficient models for multilingual phone recognitioncited by 19
  4. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 17
  5. 2024Anatomy of Industrial Scale Multilingual ASRcited by 14
  6. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  7. 2022Boosting Cross-Domain Speech Recognition with Self-Supervisioncited by 10
  8. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASRcited by 10
  9. 2023Contextual Biasing with the Knuth-Morris-Pratt Matching Algorithmcited by 9
  10. 2024Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Modelscited by 6
  11. 2024Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASRcited by 5
  12. 2023Lego-Features: Exporting modular encoder features for streaming and deliberation ASRcited by 3
13 more
  1. 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layercited by 2
  2. 2023Improved Long-Form Speech Recognition by Jointly Modeling the Primary and Non-primary Speakerscited by 2
  3. 2024Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Modelscited by 2
  4. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  5. 2025Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filteringcited by 2
  6. 2025BEST-RQ-Based Self-Supervised Learning for Whisper Domain Adaptationcited by 2
  7. 2024Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whispercited by 1
  8. 2025Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducerscited by 1
  9. 2025MSDA: Combining Pseudo-labeling and Self-Supervision for Unsupervised Domain Adaptation in ASRcited by 1
  10. 2026Gumbel-BEARD: Automatic Layer Selection for Self-Supervised Adaptation of Whisper in Low-Resource Domainscited by 1
  11. 2023Edit Distance based RL for RNNT decodingcited by 0
  12. 2026Leveraging Audio-LLMs to Filter Speech-to-Speech Training Datacited by 0
  13. 2026StreamHear: Domain-Adapted Pseudo-Labeling for Semi-Supervised Streaming Speech Recognitioncited by 0