# Dynamic Stale Synchronous Parallel Distributed Training for Deep Learning

arXiv: https://arxiv.org/abs/1908.11848 · HTML: https://mlsanity.com/paper/1908.11848/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2019.

## Built on by (most cited first, 12 of 12)

- [A Comprehensive Survey on Distributed Training of Graph Neural Networks](https://mlsanity.com/paper/2211.05368/) (2022)
- [An Interdisciplinary Outlook on Large Language Models for Scientific Research](https://mlsanity.com/paper/2311.04929/) (2023)
- [Communication-Efficient Federated Learning for Heterogeneous Edge Devices Based on Adaptive Gradient Quantization](https://mlsanity.com/paper/2212.08272/) (2022)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning](https://mlsanity.com/paper/2104.08364/) (2021)
- [OSP: Boosting Distributed Model Training with 2-stage Synchronization](https://mlsanity.com/paper/2306.16926/) (2023)
- [Fair and Efficient Distributed Edge Learning with Hybrid Multipath TCP](https://mlsanity.com/paper/2211.09723/) (2022)
- [Distributed Training under Packet Loss](https://mlsanity.com/paper/2507.07114/) (2025)
- [ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusters](https://mlsanity.com/paper/2308.15164/) (2023)
- [Dynamic Clustering for Personalized Federated Learning on Heterogeneous Edge Devices](https://mlsanity.com/paper/2508.01580/) (2025)
- [Hybrid Dual-Batch and Cyclic Progressive Learning for Efficient Distributed Training](https://mlsanity.com/paper/2509.26092/) (2025)
- [A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability](https://mlsanity.com/paper/2605.18750/) (2026)

## Abstract (arXiv)

Deep learning is a popular machine learning technique and has been applied to many real-world problems. However, training a deep neural network is very time-consuming, especially on big data. It has become difficult for a single machine to train a large model over large datasets. A popular solution is to distribute and parallelize the training process across multiple machines using the parameter server framework. In this paper, we present a distributed paradigm on the parameter server framework called Dynamic Stale Synchronous Parallel (DSSP) which improves the state-of-the-art Stale Synchronous Parallel (SSP) paradigm by dynamically determining the staleness threshold at the run time. Conventionally to run distributed training in SSP, the user needs to specify a particular staleness threshold as a hyper-parameter. However, a user does not usually know how to set the threshold and thus often finds a threshold value through trial and error, which is time-consuming. Based on workers' recent processing time, our approach DSSP adaptively adjusts the threshold per iteration at running time to reduce the waiting time of faster workers for synchronization of the globally shared parameters, and consequently increases the frequency of parameters updates (increases iteration throughput), which speedups the convergence rate. We compare DSSP with other paradigms such as Bulk Synchronous Parallel (BSP), Asynchronous Parallel (ASP), and SSP by running deep neural networks (DNN) models over GPU clusters in both homogeneous and heterogeneous environments. The results show that in a heterogeneous environment where the cluster consists of mixed models of GPUs, DSSP converges to a higher accuracy much earlier than SSP and BSP and performs similarly to ASP. In a homogeneous distributed cluster, DSSP has more stable and slightly better performance than SSP and ASP, and converges much faster than BSP.