mlsanityData to 7 Oct 2026

Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Mask

arXiv 2110.04056 ↗First posted 8 Oct 2021eess.AS cs.LG cs.SDcited by 3

Not on the map. Why?

Abstract (arXiv)

In the recent trend of semi-supervised speech recognition, both self-supervised representation learning and pseudo-labeling have shown promising results. In this paper, we propose a novel approach to combine their ideas for end-to-end speech recognition model. Without any extra loss function, we utilize the Gradient Mask to optimize the model when training on pseudo-label. This method forces the speech recognition model to predict from the masked input to learn strong acoustic representation and make training robust to label noise. In our semi-supervised experiments, the method can improve the model performance when training on pseudo-label and our method achieved competitive results comparing with other semi-supervised approaches on the Librispeech 100 hours experiments.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

25 of 36 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  7. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  8. 2015Listen, Attend and Spellcited by 542
  9. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  11. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  12. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
13 more
  1. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  2. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
  3. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  4. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  5. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  6. 2019RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentationcited by 63
  7. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  8. 2019Lessons from Building Acoustic Models with a Million Hours of Speechcited by 27
  9. 2020Joint Masked CPC and CTC Training for ASRcited by 20
  10. 2019BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognitioncited by 10
  11. 2020Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognitioncited by 9
  12. 2021UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Datasetcited by 7
  13. 2019Lead2Gold: Towards exploiting the full potential of noisy transcriptions for speech recognitioncited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 1.

3 papers.

  1. 2022Self-Training: A Surveycited by 85
  2. 2023Decoupled Structure for Improved Adaptability of End-to-End Modelscited by 3
  3. 2025Customizing Speech Recognition Model with Large Language Model Feedbackcited by 1