# Asynchronous Training Schemes in Distributed Learning with Time Delay

arXiv: https://arxiv.org/abs/2208.13154 · HTML: https://mlsanity.com/paper/2208.13154/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 28 Aug 2022.

## Builds on (most cited first, 25 of 25)

- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Asynchronous Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06952/) (2017)
- [Revisiting Distributed Synchronous SGD](https://mlsanity.com/paper/1702.05800/) (2017)
- [Distributionally Robust Federated Averaging](https://mlsanity.com/paper/2102.12660/) (2021)
- [A generic framework for privacy preserving deep learning](https://mlsanity.com/paper/1811.04017/) (2018)
- [Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation](https://mlsanity.com/paper/1903.07424/) (2019)
- [Asynchronous Stochastic Gradient Descent with Delay Compensation](https://mlsanity.com/paper/1609.08326/) (2016)
- [Collaborative Deep Learning in Fixed Topology Networks](https://mlsanity.com/paper/1706.07880/) (2017)
- [Communication-Efficient Algorithms for Decentralized and Stochastic Optimization](https://mlsanity.com/paper/1701.03961/) (2017)
- [An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimization](https://mlsanity.com/paper/1505.04824/) (2015)
- [Gossip training for deep learning](https://mlsanity.com/paper/1611.09726/) (2016)
- [Accelerated Methods for Non-Convex Optimization](https://mlsanity.com/paper/1611.00756/) (2016)
- [Fully Decoupled Neural Network Learning Using Delayed Gradients](https://mlsanity.com/paper/1906.09108/) (2019)
- [Compact and Computationally Efficient Representation of Deep Neural Networks](https://mlsanity.com/paper/1805.10692/) (2018)
- [Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data](https://mlsanity.com/paper/1905.12648/) (2019)
- [Asynchrony and Acceleration in Gossip Algorithms](https://mlsanity.com/paper/2011.02379/) (2020)
- [Efficient Distributed Online Prediction and Stochastic Optimization with Approximate Distributed Averaging](https://mlsanity.com/paper/1403.0603/) (2014)
- [Asynchronous Decentralized Learning of a Neural Network](https://mlsanity.com/paper/2004.05082/) (2020)
- [Adaptive Braking for Mitigating Gradient Delay](https://mlsanity.com/paper/2007.01397/) (2020)
- [SharpEdge: An Asynchronous and Core-Agnostic Solution to Guarantee Bounded-Delays](https://mlsanity.com/paper/2001.00112/) (2019)
- [DC-S3GD: Delay-Compensated Stale-Synchronous SGD for Large-Scale Decentralized Neural Network Training](https://mlsanity.com/paper/1911.02516/) (2019)
- [Asynchronous Consensus Algorithm](https://mlsanity.com/paper/2001.07704/) (2020)

## Built on by (most cited first, 2 of 2)

- [Straggler-Resilient Decentralized Learning via Adaptive Asynchronous Updates](https://mlsanity.com/paper/2306.06559/) (2023)
- [Asynchronous Heavy-Tailed Optimization](https://mlsanity.com/paper/2602.18002/) (2026)

## Abstract (arXiv)

In the context of distributed deep learning, the issue of stale weights or gradients could result in poor algorithmic performance. This issue is usually tackled by delay tolerant algorithms with some mild assumptions on the objective functions and step sizes. In this paper, we propose a different approach to develop a new algorithm, called Predicting Clipping Asynchronous Stochastic Gradient Descent (aka, PC-ASGD). Specifically, PC-ASGD has two steps - the textitpredicting step leverages the gradient prediction using Taylor expansion to reduce the staleness of the outdated weights while the textitclipping step selectively drops the outdated weights to alleviate their negative effects. A tradeoff parameter is introduced to balance the effects between these two steps. Theoretically, we present the convergence rate considering the effects of delay of the proposed algorithm with constant step size when the smooth objective functions are weakly strongly-convex and nonconvex. One practical variant of PC-ASGD is also proposed by adopting a condition to help with the determination of the tradeoff parameter. For empirical validation, we demonstrate the performance of the algorithm with two deep neural network architectures on two benchmark datasets.