# Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD

arXiv: https://arxiv.org/abs/1810.08313 · HTML: https://mlsanity.com/paper/1810.08313/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2018.

## Built on by (most cited first, 30 of 77)

- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients](https://mlsanity.com/paper/2102.07053/) (2021)
- [Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling](https://mlsanity.com/paper/2112.11256/) (2021)
- [Hierarchical Federated Learning with Quantization: Convergence Analysis and System Design](https://mlsanity.com/paper/2103.14272/) (2021)
- [A Unified Analysis of Federated Learning with Arbitrary Client Participation](https://mlsanity.com/paper/2205.13648/) (2022)
- [Layer-wise Adaptive Model Aggregation for Scalable Federated Learning](https://mlsanity.com/paper/2110.10302/) (2021)
- [Trading Off Privacy, Utility and Efficiency in Federated Learning](https://mlsanity.com/paper/2209.00230/) (2022)
- [Resource-Efficient Federated Learning](https://mlsanity.com/paper/2111.01108/) (2021)
- [Accelerating Gossip SGD with Periodic Global Averaging](https://mlsanity.com/paper/2105.09080/) (2021)
- [Cost-Effective Federated Learning in Mobile Edge Networks](https://mlsanity.com/paper/2109.05411/) (2021)
- [Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction](https://mlsanity.com/paper/2310.08670/) (2023)
- [FedShuffle: Recipes for Better Use of Local Work in Federated Learning](https://mlsanity.com/paper/2204.13169/) (2022)
- [Trade-offs of Local SGD at Scale: An Empirical Study](https://mlsanity.com/paper/2110.08133/) (2021)
- [Communication-efficient SGD: From Local SGD to One-Shot Averaging](https://mlsanity.com/paper/2106.04759/) (2021)
- [Fast Federated Learning by Balancing Communication Trade-Offs](https://mlsanity.com/paper/2105.11028/) (2021)
- [GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity](https://mlsanity.com/paper/2210.16402/) (2022)
- [Adversarial training in communication constrained federated learning](https://mlsanity.com/paper/2103.01319/) (2021)
- [Distributed Learning Systems with First-order Methods](https://mlsanity.com/paper/2104.05245/) (2021)
- [Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters](https://mlsanity.com/paper/2111.05897/) (2021)
- [Towards Demystifying Serverless Machine Learning Training](https://mlsanity.com/paper/2105.07806/) (2021)
- [A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guarantee](https://mlsanity.com/paper/2108.00365/) (2021)
- [LocalNewton: Reducing Communication Bottleneck for Distributed Learning](https://mlsanity.com/paper/2105.07320/) (2021)
- [Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGX](https://mlsanity.com/paper/2105.01281/) (2021)
- [Communication-Efficient Federated Learning for Heterogeneous Edge Devices Based on Adaptive Gradient Quantization](https://mlsanity.com/paper/2212.08272/) (2022)
- [BAGUA: Scaling up Distributed Learning with System Relaxations](https://mlsanity.com/paper/2107.01499/) (2021)
- [Semi-Decentralized Federated Edge Learning with Data and Device Heterogeneity](https://mlsanity.com/paper/2112.10313/) (2021)
- [Federated Learning with Flexible Control](https://mlsanity.com/paper/2212.08496/) (2022)
- [FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learning](https://mlsanity.com/paper/2108.06869/) (2021)
- [Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learning](https://mlsanity.com/paper/2210.00690/) (2022)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)

## Abstract (arXiv)

Large-scale machine learning training, in particular distributed stochastic gradient descent, needs to be robust to inherent system variability such as node straggling and random communication delays. This work considers a distributed training framework where each worker node is allowed to perform local model updates and the resulting models are averaged periodically. We analyze the true speed of error convergence with respect to wall-clock time (instead of the number of iterations), and analyze how it is affected by the frequency of averaging. The main contribution is the design of AdaComm, an adaptive communication strategy that starts with infrequent averaging to save communication delay and improve convergence speed, and then increases the communication frequency in order to achieve a low error floor. Rigorous experiments on training deep neural networks show that AdaComm can take 3 × less time than fully synchronous SGD, and still reach the same final training loss.