mlsanityData to 7 Oct 2026

Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition

arXiv 2106.08922 ↗First posted 16 Jun 2021eess.AS cs.LG cs.SDcited by 37

Not on the map. Why?

Abstract (arXiv)

Pseudo-labeling (PL) has been shown to be effective in semi-supervised automatic speech recognition (ASR), where a base model is self-trained with pseudo-labels generated from unlabeled data. While PL can be further improved by iteratively updating pseudo-labels as the model evolves, most of the previous approaches involve inefficient retraining of the model or intricate control of the label update. We present momentum pseudo-labeling (MPL), a simple yet effective strategy for semi-supervised ASR. MPL consists of a pair of online and offline models that interact and learn from each other, inspired by the mean teacher method. The online model is trained to predict pseudo-labels generated on the fly by the offline model. The offline model maintains a momentum-based moving average of the online model. MPL is performed in a single training process and the interaction between the two models effectively helps them reinforce each other to improve the ASR performance. We apply MPL to an end-to-end ASR model based on the connectionist temporal classification. The experimental results demonstrate that MPL effectively improves over the base model and is scalable to different semi-supervised scenarios with varying amounts of data or domain mismatch.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

34 of 49 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  4. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  5. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  6. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  9. 2015Listen, Attend and Spellcited by 542
  10. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  11. 2015Attention-Based Models for Speech Recognitioncited by 400
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
22 more
  1. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  2. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  3. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  4. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  5. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  6. 2019QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutionscited by 93
  7. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  8. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
  9. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
  10. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  11. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  12. 2019RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentationcited by 63
  13. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  14. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 46
  15. 2019Almost Unsupervised Text to Speech and Automatic Speech Recognitioncited by 41
  16. 2020Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Trainingcited by 38
  17. 2018Cycle-consistency training for end-to-end speech recognitioncited by 31
  18. 2020Semi-supervised ASR by End-to-end Self-trainingcited by 29
  19. 2020Semi-Supervised Learning with Data Augmentation for End-to-End ASRcited by 19
  20. 2020Semi-Supervised Speech Recognition via Graph-based Temporal Classificationcited by 19
  21. 2020Semi-Supervised Speech Recognition via Local Prior Matchingcited by 13
  22. 2019Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Textcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 4; 2024: 10; 2025: 1; 2026: 5.

37 papers.

Sort:

By area:

  1. 2022Semi-supervised Vision Transformers at Scalecited by 49
  2. 2021Pseudo-Labeling for Massively Multilingual Speech Recognitioncited by 23
  3. 2022RemixIT: Continual self-training of speech enhancement models via bootstrapped remixingcited by 20
  4. 2024CR-CTC: Consistency regularization on CTC for improved speech recognitioncited by 15
  5. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 14
  6. 2024Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Surveycited by 12
  7. 2022Continuous Pseudo-Labeling from the Startcited by 11
  8. 2022Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessmentcited by 10
  9. 2022Boosting Cross-Domain Speech Recognition with Self-Supervisioncited by 10
  10. 2021Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASRcited by 7
  11. 2022Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greekcited by 7
  12. 2023A Novel Self-training Approach for Low-resource Speech Recognitioncited by 7
25 more
  1. 2021Continual self-training with bootstrapped remixing for speech enhancementcited by 6
  2. 2022Decoupled Federated Learning for ASR with Non-IID Datacited by 6
  3. 2022Continuous Soft Pseudo-Labeling in ASRcited by 6
  4. 2023Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognitioncited by 6
  5. 2023AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognitioncited by 5
  6. 2022Filter and evolve: progressive pseudo label refining for semi-supervised automatic speech recognitioncited by 4
  7. 2024Conformer-1: Robust ASR via Large-Scale Semisupervised Bootstrappingcited by 4
  8. 2021Word Order Does Not Matter For Speech Recognitioncited by 2
  9. 2022Improved Consistency Training for Semi-Supervised Sequence-to-Sequence ASR via Speech Chain Reconstruction and Self-Transcribingcited by 2
  10. 2022More Speaking or More Speakers?cited by 2
  11. 2024Token-Weighted RNN-T for Learning from Flawed Datacited by 2
  12. 2024Is user feedback always informative? Retrieval Latent Defending for Semi-Supervised Domain Adaptation without Source Datacited by 2
  13. 2024Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whispercited by 1
  14. 2024Bridging the Gaps: Utilizing Unlabeled Face Recognition Datasets to Boost Semi-Supervised Facial Expression Recognitioncited by 1
  15. 2026ReHear: Iterative Pseudo-Label Refinement for Semi-Supervised Speech Recognition via Audio Large Language Modelscited by 1
  16. 2026Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognitioncited by 1
  17. 2022Direction-Aware Joint Adaptation of Neural Speech Enhancement and Recognition in Real Multiparty Conversational Environmentscited by 0
  18. 2022InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Losscited by 0
  19. 2023Unsupervised ASR via Cross-Lingual Pseudo-Labelingcited by 0
  20. 2024Self-Train Before You Transcribecited by 0
  21. 2024Transliterated Zero-Shot Domain Adaptation for Automatic Speech Recognitioncited by 0
  22. 2025LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Datacited by 0
  23. 2026StreamHear: Domain-Adapted Pseudo-Labeling for Semi-Supervised Streaming Speech Recognitioncited by 0
  24. 2026A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilizationcited by 0
  25. 2026Pretrained ASR Pseudo-labeling for Noisy Police Audiocited by 0