mlsanityData to 14 Oct 2026

SLowcal-SGD: Slow Query Points Improve Local-SGD for Stochastic Convex Optimization

arXiv 2304.04169 ↗First posted 9 Apr 2023cs.LG math.OCcited by 4

Not on the map. Why?

Abstract (arXiv)

We consider distributed learning scenarios where M machines interact with a parameter server along several communication rounds in order to minimize a joint objective function. Focusing on the heterogeneous case, where different machines may draw samples from different data-distributions, we design the first local update method that provably benefits over the two most prominent distributed baselines: namely Minibatch-SGD and Local-SGD. Key to our approach is a slow querying technique that we customize to the distributed setting, which in turn enables a better mitigation of the bias caused by local updates.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 9 Apr 2023.

Builds on

22 of 40 references.

References · 22

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  4. 2019Introduction to Online Convex Optimizationcited by 803
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  6. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  7. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  8. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  9. 2021A Field Guide to Federated Optimizationcited by 315
  10. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  11. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  12. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
10 more
  1. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  2. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  3. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  4. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  5. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  6. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  7. 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
  8. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  9. 2021No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimizationcited by 20
  10. 2020Adaptive Gradient Methods for Constrained Convex Optimization and Variational Inequalitiescited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 2; 2026: 1.

4 papers.

  1. 2023Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysiscited by 11
  2. 2025Enhancing Parallelism in Decentralized Stochastic Convex Optimizationcited by 1
  3. 2025Beyond Communication Overhead: A Multilevel Monte Carlo Approach for Mitigating Compression Bias in Distributed Learningcited by 0
  4. 2026Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learningcited by 0