# Convex and Non-convex Federated Learning with Stale Stochastic Gradients: Diminishing Step Size is All You Need

arXiv: https://arxiv.org/abs/2603.02639 · HTML: https://mlsanity.com/paper/2603.02639/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Builds on (most cited first, 13 of 13)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Handbook of Convergence Theorems for (Stochastic) Gradient Methods](https://mlsanity.com/paper/2301.11235/) (2023)
- [Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimization](https://mlsanity.com/paper/1308.6594/) (2013)
- [An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback](https://mlsanity.com/paper/1507.08752/) (2015)
- [Distributed optimization over time-varying directed graphs](https://mlsanity.com/paper/1303.2289/) (2013)
- [Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning](https://mlsanity.com/paper/2201.09531/) (2022)
- [Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity](https://mlsanity.com/paper/2012.14453/) (2020)
- [Non-Convex Distributed Optimization](https://mlsanity.com/paper/1512.00895/) (2015)
- [Zeroth-Order Feedback Optimization for Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2011.09728/) (2020)
- [Distributed Optimization over Time-varying Graphs with Imperfect Sharing of Information](https://mlsanity.com/paper/2106.08469/) (2021)
- [DIMIX: DIminishing MIXing for Sloppy Agents](https://mlsanity.com/paper/2204.10974/) (2022)
- [Zeroth-Order Non-Convex Optimization for Cooperative Multi-Agent Systems with Diminishing Step Size and Smoothing Radius](https://mlsanity.com/paper/2401.03998/) (2024)

## Abstract (arXiv)

We propose a general framework for distributed stochastic optimization under delayed gradient models. In this setting, n local agents leverage their own data and computation to assist a central server in minimizing a global objective composed of agents' local cost functions. Each agent is allowed to transmit stochastic-potentially biased and delayed-estimates of its local gradient. While a prior work has advocated delay-adaptive step sizes for stochastic gradient descent (SGD) in the presence of delays, we demonstrate that a pre-chosen diminishing step size is sufficient and matches the performance of the adaptive scheme. Moreover, our analysis establishes that diminishing step sizes recover the optimal SGD rates for nonconvex and strongly convex objectives.