# Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study

arXiv: https://arxiv.org/abs/1509.04210 · HTML: https://mlsanity.com/paper/1509.04210/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2015.

## Built on by (most cited first, 19 of 19)

- [SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks](https://mlsanity.com/paper/2106.02743/) (2021)
- [AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregation](https://mlsanity.com/paper/2205.13797/) (2022)
- [An Invitation to Distributed Quantum Neural Networks](https://mlsanity.com/paper/2211.07056/) (2022)
- [ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table](https://mlsanity.com/paper/2104.08542/) (2021)
- [Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching](https://mlsanity.com/paper/2305.12213/) (2023)
- [SubStrat: A Subset-Based Strategy for Faster AutoML](https://mlsanity.com/paper/2206.03070/) (2022)
- [Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training](https://mlsanity.com/paper/2303.06659/) (2023)
- [Hierarchical Federated ADMM](https://mlsanity.com/paper/2409.18796/) (2024)
- [Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent](https://mlsanity.com/paper/2112.01433/) (2021)
- [Asynchronous Decentralized Distributed Training of Acoustic Models](https://mlsanity.com/paper/2110.11199/) (2021)
- [STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronization](https://mlsanity.com/paper/2210.03521/) (2022)
- [Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging](https://mlsanity.com/paper/2405.20988/) (2024)
- [Scheduling Optimization Techniques for Neural Network Training](https://mlsanity.com/paper/2110.00929/) (2021)
- [Quantized Adaptive Subgradient Algorithms and Their Applications](https://mlsanity.com/paper/2208.05631/) (2022)
- [DRAG: Divergence-based Adaptive Aggregation in Federated learning on Non-IID Data](https://mlsanity.com/paper/2309.01779/) (2023)
- [System-level Impact of Non-Ideal Program-Time of Charge Trap Flash (CTF) on Deep Neural Network](https://mlsanity.com/paper/2402.09792/) (2024)
- [A Hierarchical Federated Learning Approach for the Internet of Things](https://mlsanity.com/paper/2403.01540/) (2024)
- [Dropout-Robust Mechanisms for Differentially Private and Fully Decentralized Mean Estimation](https://mlsanity.com/paper/2506.03746/) (2025)
- [Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning](https://mlsanity.com/paper/2511.00655/) (2025)

## Abstract (arXiv)

This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtime performance and model accuracy. We introduce a new learning rate modulation strategy to counter the effect of stale gradients and propose a new synchronization protocol that can effectively bound the staleness in gradients, improve runtime performance and achieve good model accuracy. Our empirical investigation reveals a principled approach for distributed training of neural networks: the mini-batch size per learner should be reduced as more learners are added to the system to preserve the model accuracy. We validate this approach using commonly-used image classification benchmarks: CIFAR10 and ImageNet.