# Experiments on Parallel Training of Deep Neural Network using Model Averaging

arXiv: https://arxiv.org/abs/1507.01239 · HTML: https://mlsanity.com/paper/1507.01239/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2015.

## Built on by (most cited first, 14 of 14)

- [Why (and When) does Local SGD Generalize Better than SGD?](https://mlsanity.com/paper/2303.01215/) (2023)
- [PopulAtion Parameter Averaging (PAPA)](https://mlsanity.com/paper/2304.03094/) (2023)
- [Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexity](https://mlsanity.com/paper/2102.04635/) (2021)
- [Trade-offs of Local SGD at Scale: An Empirical Study](https://mlsanity.com/paper/2110.08133/) (2021)
- [Training Neural Networks from Scratch with Parallel Low-Rank Adapters](https://mlsanity.com/paper/2402.16828/) (2024)
- [An Experimental Study of Data Heterogeneity in Federated Learning Methods for Medical Imaging](https://mlsanity.com/paper/2107.08371/) (2021)
- [Handling Data Heterogeneity with Generative Replay in Collaborative Learning for Medical Imaging](https://mlsanity.com/paper/2106.13208/) (2021)
- [Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks](https://mlsanity.com/paper/2101.05014/) (2021)
- [Federated Compositional Deep AUC Maximization](https://mlsanity.com/paper/2304.10101/) (2023)
- [A Quadratic Synchronization Rule for Distributed Deep Learning](https://mlsanity.com/paper/2310.14423/) (2023)
- [ParamDelta for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost](https://mlsanity.com/paper/2504.21023/) (2025)
- [CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Manner](https://mlsanity.com/paper/2103.07974/) (2021)
- [Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning](https://mlsanity.com/paper/2507.20424/) (2025)
- [Substituting Proof of Work in Blockchain with Training-Verified Collaborative Model Computation](https://mlsanity.com/paper/2508.12138/) (2025)

## Abstract (arXiv)

In this work we apply model averaging to parallel training of deep neural network (DNN). Parallelization is done in a model averaging manner. Data is partitioned and distributed to different nodes for local model updates, and model averaging across nodes is done every few minibatches. We use multiple GPUs for data parallelization, and Message Passing Interface (MPI) for communication between nodes, which allows us to perform model averaging frequently without losing much time on communication. We investigate the effectiveness of Natural Gradient Stochastic Gradient Descent (NG-SGD) and Restricted Boltzmann Machine (RBM) pretraining for parallel training in model-averaging framework, and explore the best setups in term of different learning rate schedules, averaging frequencies and minibatch sizes. It is shown that NG-SGD and RBM pretraining benefits parameter-averaging based model training. On the 300h Switchboard dataset, a 9.3 times speedup is achieved using 16 GPUs and 17 times speedup using 32 GPUs with limited decoding accuracy loss.