# Asynchronous Decentralized Distributed Training of Acoustic Models

arXiv: https://arxiv.org/abs/2110.11199 · HTML: https://mlsanity.com/paper/2110.11199/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 21 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Asynchronous Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06952/) (2017)
- [On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimization](https://mlsanity.com/paper/1708.01012/) (2017)
- [Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://mlsanity.com/paper/1807.11205/) (2018)
- [English Conversational Telephone Speech Recognition by Humans and Machines](https://mlsanity.com/paper/1703.02136/) (2017)
- [Lessons from Building Acoustic Models with a Million Hours of Speech](https://mlsanity.com/paper/1904.01624/) (2019)
- [Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study](https://mlsanity.com/paper/1509.04210/) (2015)
- [Scale MLPerf-0.6 models on Google TPU-v3 Pods](https://mlsanity.com/paper/1909.09756/) (2019)
- [Realizing Petabyte Scale Acoustic Modeling](https://mlsanity.com/paper/1904.10584/) (2019)
- [A Highly Efficient Distributed Deep Learning System For Automatic Speech Recognition](https://mlsanity.com/paper/1907.05701/) (2019)
- [Distributed Deep Learning Strategies For Automatic Speech Recognition](https://mlsanity.com/paper/1904.04956/) (2019)
- [Improving Efficiency in Large-Scale Decentralized Distributed Training](https://mlsanity.com/paper/2002.01119/) (2020)
- [Sequence Training of DNN Acoustic Models With Natural Gradient](https://mlsanity.com/paper/1804.02204/) (2018)
- [A Random Gossip BMUF Process for Neural Language Modeling](https://mlsanity.com/paper/1909.09010/) (2019)
- [Distributed Training of Deep Neural Network Acoustic Models for Automatic Speech Recognition](https://mlsanity.com/paper/2002.10502/) (2020)
- [A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Training](https://mlsanity.com/paper/2103.07554/) (2021)

## Built on by (most cited first, 2 of 2)

- [Ravnest: Decentralized Asynchronous Training on Heterogeneous Devices](https://mlsanity.com/paper/2401.01728/) (2024)
- [A Treatise On FST Lattice Based MMI Training](https://mlsanity.com/paper/2210.08918/) (2022)

## Abstract (arXiv)

Large-scale distributed training of deep acoustic models plays an important role in today's high-performance automatic speech recognition (ASR). In this paper we investigate a variety of asynchronous decentralized distributed training strategies based on data parallel stochastic gradient descent (SGD) to show their superior performance over the commonly-used synchronous distributed training via allreduce, especially when dealing with large batch sizes. Specifically, we study three variants of asynchronous decentralized parallel SGD (ADPSGD), namely, fixed and randomized communication patterns on a ring as well as a delay-by-one scheme. We introduce a mathematical model of ADPSGD, give its theoretical convergence rate, and compare the empirical convergence behavior and straggler resilience properties of the three variants. Experiments are carried out on an IBM supercomputer for training deep long short-term memory (LSTM) acoustic models on the 2000-hour Switchboard dataset. Recognition and speedup performance of the proposed strategies are evaluated under various training configurations. We show that ADPSGD with fixed and randomized communication patterns cope well with slow learners. When learners are equally fast, ADPSGD with the delay-by-one strategy has the fastest convergence with large batches. In particular, using the delay-by-one strategy, we can train the acoustic model in less than 2 hours using 128 V100 GPUs with competitive word error rates.