# Communication trade-offs for synchronized distributed SGD with large step size

arXiv: https://arxiv.org/abs/1904.11325 · HTML: https://mlsanity.com/paper/1904.11325/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2019.

## Built on by (most cited first, 4 of 4)

- [FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction](https://mlsanity.com/paper/2203.11751/) (2022)
- [Towards Federated Learning on Time-Evolving Heterogeneous Data](https://mlsanity.com/paper/2112.13246/) (2021)
- [A Convergence Theory for Federated Average: Beyond Smoothness](https://mlsanity.com/paper/2211.01588/) (2022)
- [Convergence Analysis of Federated Learning Methods Using Backward Error Analysis](https://mlsanity.com/paper/2503.03139/) (2025)

## Abstract (arXiv)

Synchronous mini-batch SGD is state-of-the-art for large-scale distributed machine learning. However, in practice, its convergence is bottlenecked by slow communication rounds between worker nodes. A natural solution to reduce communication is to use the `local-SGD' model in which the workers train their model independently and synchronize every once in a while. This algorithm improves the computation-communication trade-off but its convergence is not understood very well. We propose a non-asymptotic error analysis, which enables comparison to one-shot averaging i.e., a single communication round among independent workers, and mini-batch averaging i.e., communicating at every step. We also provide adaptive lower bounds on the communication frequency for large step-sizes ( t-α ,  α∈ (1/2 , 1 ) ) and show that Local-SGD reduces communication by a factor of OBig(frac√TP3/2Big), with T the total number of gradients and P machines.