# STL-SGD: Speeding Up Local SGD with Stagewise Communication Period

arXiv: https://arxiv.org/abs/2006.06377 · HTML: https://mlsanity.com/paper/2006.06377/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 8 of 8)

- [A Quadratic Synchronization Rule for Distributed Deep Learning](https://mlsanity.com/paper/2310.14423/) (2023)
- [Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysis](https://mlsanity.com/paper/2501.04443/) (2025)
- [The Role of Local Steps in Local SGD](https://mlsanity.com/paper/2203.06798/) (2022)
- [STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronization](https://mlsanity.com/paper/2210.03521/) (2022)
- [EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models](https://mlsanity.com/paper/2412.07210/) (2024)
- [Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning](https://mlsanity.com/paper/2507.20424/) (2025)
- [Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems](https://mlsanity.com/paper/2508.08540/) (2025)
- [AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization](https://mlsanity.com/paper/2609.36662/) (2026)

## Abstract (arXiv)

Distributed parallel stochastic gradient descent algorithms are workhorses for large scale machine learning tasks. Among them, local stochastic gradient descent (Local SGD) has attracted significant attention due to its low communication complexity. Previous studies prove that the communication complexity of Local SGD with a fixed or an adaptive communication period is in the order of O (N^frac32 T^frac12) and O (N^frac34 T^frac34) when the data distributions on clients are identical (IID) or otherwise (Non-IID), where N is the number of clients and T is the number of iterations. In this paper, to accelerate the convergence by reducing the communication complexity, we propose textitSTagewise textitLocal textitSGD (STL-SGD), which increases the communication period gradually along with decreasing learning rate. We prove that STL-SGD can keep the same convergence rate and linear speedup as mini-batch SGD. In addition, as the benefit of increasing the communication period, when the objective is strongly convex or satisfies the Polyak-L ojasiewicz condition, the communication complexity of STL-SGD is O (N logT) and O (N^frac12 T^frac12) for the IID case and the Non-IID case respectively, achieving significant improvements over Local SGD. Experiments on both convex and non-convex problems demonstrate the superior performance of STL-SGD.