# Parallel training of DNNs with Natural Gradient and Parameter Averaging

arXiv: https://arxiv.org/abs/1410.7455 · HTML: https://mlsanity.com/paper/1410.7455/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2014.

## Built on by (most cited first, 11 of 11)

- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning](https://mlsanity.com/paper/2207.04338/) (2022)
- [Why (and When) does Local SGD Generalize Better than SGD?](https://mlsanity.com/paper/2303.01215/) (2023)
- [Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexity](https://mlsanity.com/paper/2102.04635/) (2021)
- [Kronecker-factored Quasi-Newton Methods for Deep Learning](https://mlsanity.com/paper/2102.06737/) (2021)
- [Comparing CTC and LFMMI for out-of-domain adaptation of wav2vec 2.0 acoustic model](https://mlsanity.com/paper/2104.02558/) (2021)
- [Certified Robustness in Federated Learning](https://mlsanity.com/paper/2206.02535/) (2022)
- [End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting](https://mlsanity.com/paper/2207.08179/) (2022)
- [Using Kaldi for Automatic Speech Recognition of Conversational Austrian German](https://mlsanity.com/paper/2301.06475/) (2023)
- [A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Training](https://mlsanity.com/paper/2103.07554/) (2021)
- [Acoustic Model Optimization over Multiple Data Sources: Merging and Valuation](https://mlsanity.com/paper/2410.15620/) (2024)

## Abstract (arXiv)

We describe the neural-network training framework used in the Kaldi speech recognition toolkit, which is geared towards training DNNs with large amounts of training data using multiple GPU-equipped or multi-core machines. In order to be as hardware-agnostic as possible, we needed a way to use multiple machines without generating excessive network traffic. Our method is to average the neural network parameters periodically (typically every minute or two), and redistribute the averaged parameters to the machines for further training. Each machine sees different data. By itself, this method does not work very well. However, we have another method, an approximate and efficient implementation of Natural Gradient for Stochastic Gradient Descent (NG-SGD), which seems to allow our periodic-averaging method to work well, as well as substantially improving the convergence of SGD on a single machine.