# Adaptive Worker Grouping For Communication-Efficient and Straggler-Tolerant Distributed SGD

arXiv: https://arxiv.org/abs/2201.04301 · HTML: https://mlsanity.com/paper/2201.04301/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 12 Jan 2022.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning](https://mlsanity.com/paper/1805.09965/) (2018)
- [Communication-Computation Efficient Gradient Coding](https://mlsanity.com/paper/1802.03475/) (2018)
- [ErasureHead: Distributed Gradient Descent without Delays Using Approximate Gradient Coding](https://mlsanity.com/paper/1901.09671/) (2019)
- [A Fundamental Tradeoff between Computation and Communication in Distributed Computing](https://mlsanity.com/paper/1604.07086/) (2016)
- [Gradient Coding with Clustering and Multi-message Communication](https://mlsanity.com/paper/1903.01974/) (2019)
- [LAGC: Lazily Aggregated Gradient Coding for Straggler-Tolerant and Communication-Efficient Distributed Learning](https://mlsanity.com/paper/1905.09148/) (2019)
- [CADA: Communication-Adaptive Distributed Adam](https://mlsanity.com/paper/2012.15469/) (2020)
- [Coded MapReduce](https://mlsanity.com/paper/1512.01625/) (2015)
- [Rateless Codes for Near-Perfect Load Balancing in Distributed Matrix-Vector Multiplication](https://mlsanity.com/paper/1804.10331/) (2018)
- [LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning](https://mlsanity.com/paper/2002.11360/) (2020)

## Abstract (arXiv)

Wall-clock convergence time and communication load are key performance metrics for the distributed implementation of stochastic gradient descent (SGD) in parameter server settings. Communication-adaptive distributed Adam (CADA) has been recently proposed as a way to reduce communication load via the adaptive selection of workers. CADA is subject to performance degradation in terms of wall-clock convergence time in the presence of stragglers. This paper proposes a novel scheme named grouping-based CADA (G-CADA) that retains the advantages of CADA in reducing the communication load, while increasing the robustness to stragglers at the cost of additional storage at the workers. G-CADA partitions the workers into groups of workers that are assigned the same data shards. Groups are scheduled adaptively at each iteration, and the server only waits for the fastest worker in each selected group. We provide analysis and experimental results to elaborate the significant gains on the wall-clock time, as well as communication load and computation load, of G-CADA over other benchmark schemes.