# Noisy student-teacher training for robust keyword spotting

arXiv: https://arxiv.org/abs/2106.01604 · HTML: https://mlsanity.com/paper/2106.01604/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [Learning to Self-Train for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1906.00562/) (2019)
- [Automatic adaptation of object detectors to new domains using self-training](https://mlsanity.com/paper/1904.07305/) (2019)
- [SpecAugment on Large Scale Datasets](https://mlsanity.com/paper/1912.05533/) (2019)
- [End-to-End Streaming Keyword Spotting](https://mlsanity.com/paper/1812.02802/) (2018)
- [Streaming Small-Footprint Keyword Spotting using Sequence-to-Sequence Models](https://mlsanity.com/paper/1710.09617/) (2017)
- [A Cascade Architecture for Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1712.03603/) (2017)
- [Learning To Detect Keyword Parts And Whole By Smoothed Max Pooling](https://mlsanity.com/paper/2001.09246/) (2020)
- [Semi-supervised and Population Based Training for Voice Commands Recognition](https://mlsanity.com/paper/1905.04230/) (2019)

## Built on by (most cited first, 10 of 10)

- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Production federated keyword spotting via distillation, filtering, and joint federated-centralized training](https://mlsanity.com/paper/2204.06322/) (2022)
- [Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensors](https://mlsanity.com/paper/2408.12481/) (2024)
- [Low-resource Low-footprint Wake-word Detection using Knowledge Distillation](https://mlsanity.com/paper/2207.03331/) (2022)
- [Locale Encoding For Scalable Multilingual Keyword Spotting Models](https://mlsanity.com/paper/2302.12961/) (2023)
- [Speech Augmentation Based Unsupervised Learning for Keyword Spotting](https://mlsanity.com/paper/2205.14329/) (2022)
- [Spot keywords from very noisy and mixed speech](https://mlsanity.com/paper/2305.17706/) (2023)
- [Personalizing Keyword Spotting with Speaker Information](https://mlsanity.com/paper/2311.03419/) (2023)
- [Adaptive Noise Resilient Keyword Spotting Using One-Shot Learning](https://mlsanity.com/paper/2505.09304/) (2025)
- [Contrastive Augmentation: An Unsupervised Learning Approach for Keyword Spotting in Speech Technology](https://mlsanity.com/paper/2409.00356/) (2024)

## Abstract (arXiv)

We propose self-training with noisy student-teacher approach for streaming keyword spotting, that can utilize large-scale unlabeled data and aggressive data augmentation. The proposed method applies aggressive data augmentation (spectral augmentation) on the input of both student and teacher and utilize unlabeled data at scale, which significantly boosts the accuracy of student against challenging conditions. Such aggressive augmentation usually degrades model performance when used with supervised training with hard-labeled data. Experiments show that aggressive spec augmentation on baseline supervised training method degrades accuracy, while the proposed self-training with noisy student-teacher training improves accuracy of some difficult-conditioned test sets by as much as 60%.