# Lessons from Building Acoustic Models with a Million Hours of Speech

arXiv: https://arxiv.org/abs/1904.01624 · HTML: https://mlsanity.com/paper/1904.01624/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 27 of 27)

- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [On the Use of External Data for Spoken Named Entity Recognition](https://mlsanity.com/paper/2112.07648/) (2021)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [Enhancing ASR for Stuttered Speech with Limited Data Using Detect and Pass](https://mlsanity.com/paper/2202.05396/) (2022)
- [Construction of a Large-scale Japanese ASR Corpus on TV Recordings](https://mlsanity.com/paper/2103.14736/) (2021)
- [Nominal Metaphor Generation with Multitask Learning](https://mlsanity.com/paper/2206.05195/) (2022)
- [Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models](https://mlsanity.com/paper/2106.06126/) (2021)
- [Training-Free Unsupervised Prompt for Vision-Language Models](https://mlsanity.com/paper/2404.16339/) (2024)
- [ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale](https://mlsanity.com/paper/2207.09078/) (2022)
- [Industry Scale Semi-Supervised Learning for Natural Language Understanding](https://mlsanity.com/paper/2103.15871/) (2021)
- [Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Mask](https://mlsanity.com/paper/2110.04056/) (2021)
- [Asynchronous Decentralized Distributed Training of Acoustic Models](https://mlsanity.com/paper/2110.11199/) (2021)
- [Combining Unsupervised and Text Augmented Semi-Supervised Learning for Low Resourced Autoregressive Speech Recognition](https://mlsanity.com/paper/2110.15836/) (2021)
- [Improving noisy student training for low-resource languages in End-to-End ASR using CycleGAN and inter-domain losses](https://mlsanity.com/paper/2407.21061/) (2024)
- [Factual Consistency Oriented Speech Recognition](https://mlsanity.com/paper/2302.12369/) (2023)
- [Self-Distillation for Gaussian Process Regression and Classification](https://mlsanity.com/paper/2304.02641/) (2023)
- [An Effective Training Framework for Light-Weight Automatic Speech Recognition Models](https://mlsanity.com/paper/2505.16991/) (2025)
- [WavFT: Acoustic model finetuning with labelled and unlabelled data](https://mlsanity.com/paper/2204.00348/) (2022)
- [Wakeword Detection under Distribution Shifts](https://mlsanity.com/paper/2207.06423/) (2022)
- [Training Autoregressive Speech Recognition Models with Limited in-domain Supervision](https://mlsanity.com/paper/2210.15135/) (2022)
- [MobileASR: A resource-aware on-device learning framework for user voice personalization applications on mobile phones](https://mlsanity.com/paper/2306.09384/) (2023)
- [An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems](https://mlsanity.com/paper/2409.10515/) (2024)

## Abstract (arXiv)

This is a report of our lessons learned building acoustic models from 1 Million hours of unlabeled speech, while labeled speech is restricted to 7,000 hours. We employ student/teacher training on unlabeled data, helping scale out target generation in comparison to confidence model based methods, which require a decoder and a confidence model. To optimize storage and to parallelize target generation, we store high valued logits from the teacher model. Introducing the notion of scheduled learning, we interleave learning on unlabeled and labeled data. To scale distributed training across a large number of GPUs, we use BMUF with 64 GPUs, while performing sequence training only on labeled data with gradient threshold compression SGD using 16 GPUs. Our experiments show that extremely large amounts of data are indeed useful; with little hyper-parameter tuning, we obtain relative WER improvements in the 10 to 20% range, with higher gains in noisier conditions.