# Improved Noisy Student Training for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2005.09629 · HTML: https://mlsanity.com/paper/2005.09629/

## Key facts

- Cited by 141 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 30 of 141)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning](https://mlsanity.com/paper/2405.00451/) (2024)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM](https://mlsanity.com/paper/2305.15255/) (2023)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [CVSS Corpus and Massively Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2201.03713/) (2022)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [Poisoning the Unlabeled Dataset of Semi-Supervised Learning](https://mlsanity.com/paper/2105.01622/) (2021)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://mlsanity.com/paper/2104.06678/) (2021)
- [GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement](https://mlsanity.com/paper/2406.11546/) (2024)
- [Pseudo-labeling for Scalable 3D Object Detection](https://mlsanity.com/paper/2103.02093/) (2021)
- [Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learning](https://mlsanity.com/paper/2110.00165/) (2021)
- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)
- [Pseudo Label Is Better Than Human Label](https://mlsanity.com/paper/2203.12668/) (2022)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [Pseudo-Labeling for Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2111.00161/) (2021)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition](https://mlsanity.com/paper/2106.07759/) (2021)
- [Improved Language Identification Through Cross-Lingual Self-Supervised Learning](https://mlsanity.com/paper/2107.04082/) (2021)

## Abstract (arXiv)

Recently, a semi-supervised learning method known as "noisy student training" has been shown to improve image classification performance of deep networks significantly. Noisy student training is an iterative self-training method that leverages augmentation to improve network performance. In this work, we adapt and improve noisy student training for automatic speech recognition, employing (adaptive) SpecAugment as the augmentation method. We find effective methods to filter, balance and augment the data generated in between self-training iterations. By doing so, we are able to obtain word error rates (WERs) 4.2%/8.6% on the clean/noisy LibriSpeech test sets by only using the clean 100h subset of LibriSpeech as the supervised set and the rest (860h) as the unlabeled set. Furthermore, we are able to achieve WERs 1.7%/3.4% on the clean/noisy LibriSpeech test sets by using the unlab-60k subset of LibriLight as the unlabeled set for LibriSpeech 960h. We are thus able to improve upon the previous state-of-the-art clean/noisy test WERs achieved on LibriSpeech 100h (4.74%/12.20%) and LibriSpeech (1.9%/4.1%).