mlsanityData to 7 Oct 2026

Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models

arXiv 2106.06126 ↗First posted 11 Jun 2021cs.SD cs.LG eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

We present results from Alexa speech teams on semi-supervised learning (SSL) of acoustic models (AM) with experiments spanning over 3000 hours of GPU time, making our study one of the largest of its kind. We discuss SSL for AMs in a small footprint setting, showing that a smaller capacity model trained with 1 million hours of unsupervised data can outperform a baseline supervised system by 14.3% word error rate reduction (WERR). When increasing the supervised data to seven-fold, our gains diminish to 7.1% WERR; to improve SSL efficiency at larger supervised data regimes, we employ a step-wise distillation into a smaller model, obtaining a WERR of 14.4%. We then switch to SSL using larger student models in low data regimes; while learning efficiency with unsupervised data is higher, student models may outperform teacher models in such a setting. We develop a theoretical sketch to explain this behavior.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 16 of its 42 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

16 of 42 references.

References · 16

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  4. 2013Do Deep Nets Really Need to be Deep?cited by 466
  5. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  6. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  7. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  8. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  9. 2020Semi-supervised ASR by End-to-end Self-trainingcited by 29
  10. 2019Lessons from Building Acoustic Models with a Million Hours of Speechcited by 27
  11. 2015Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognitioncited by 26
  12. 2020Semi-Supervised Learning with Data Augmentation for End-to-End ASRcited by 19
4 more
  1. 2019Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillationcited by 16
  2. 2018Sequence-Level Knowledge Distillation for Model Compression of Attention-based Sequence-to-Sequence Speech Recognitioncited by 9
  3. 2020High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Modelcited by 8
  4. 2019Realizing Petabyte Scale Acoustic Modelingcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 1; 2024: 1.

5 papers.

  1. 2022E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASRcited by 17
  2. 2021CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognitioncited by 8
  3. 2021Noisy Training Improves E2E ASR for the Edgecited by 2
  4. 2023Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognitioncited by 2
  5. 2024Task Integration Distillation for Object Detectorscited by 1