# Realizing Petabyte Scale Acoustic Modeling

arXiv: https://arxiv.org/abs/1904.10584 · HTML: https://mlsanity.com/paper/1904.10584/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2019.

## Built on by (most cited first, 6 of 6)

- [ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training](https://mlsanity.com/paper/2104.11125/) (2021)
- [Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models](https://mlsanity.com/paper/2106.06126/) (2021)
- [Asynchronous Decentralized Distributed Training of Acoustic Models](https://mlsanity.com/paper/2110.11199/) (2021)
- [Soft Random Sampling: A Theoretical and Empirical Analysis](https://mlsanity.com/paper/2311.12727/) (2023)
- [DROP: Poison Dilution via Knowledge Distillation for Federated Learning](https://mlsanity.com/paper/2502.07011/) (2025)
- [Sequence-level self-learning with multiple hypotheses](https://mlsanity.com/paper/2112.05826/) (2021)

## Abstract (arXiv)

Large scale machine learning (ML) systems such as the Alexa automatic speech recognition (ASR) system continue to improve with increasing amounts of manually transcribed training data. Instead of scaling manual transcription to impractical levels, we utilize semi-supervised learning (SSL) to learn acoustic models (AM) from the vast firehose of untranscribed audio data. Learning an AM from 1 Million hours of audio presents unique ML and system design challenges. We present the design and evaluation of a highly scalable and resource efficient SSL system for AM. Employing the student/teacher learning paradigm, we focus on the student learning subsystem: a scalable and robust data pipeline that generates features and targets from raw audio, and an efficient model pipeline, including the distributed trainer, that builds a student model. Our evaluations show that, even without extensive hyper-parameter tuning, we obtain relative accuracy improvements in the 10 to 20% range, with higher gains in noisier conditions. The end-to-end processing time of this SSL system was 12 days, and several components in this system can trivially scale linearly with more compute resources.