mlsanityData to 7 Oct 2026

Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning

arXiv 2606.01128 ↗First posted 31 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, and their accelerated variants, they still exhibit communication-round complexity that scales with the total number of samples N. In this paper, we introduce Local MixVR, a distributed framework that integrates local updates with variance-reduction techniques to mitigate local noise. We show that Local MixVR is the first distributed method to eliminate the dependence of communication complexity on N, achieving a complexity that scales only with the number of workers M. In common regimes where M<Oleft(N1/4right), Local MixVR outperforms the state-of-the-art Minibatch Accelerated SGD baseline, bridging a long-standing gap in distributed optimization and establishing a new paradigm for communication-efficient training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

29 of 43 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  10. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  11. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  12. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
17 more
  1. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  2. 2019A Survey on Distributed Machine Learningcited by 187
  3. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  4. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  5. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  6. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  7. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  8. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  9. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  10. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  11. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  12. 2021Proximal and Federated Random Reshufflingcited by 33
  13. 2021The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communicationcited by 30
  14. 2024INTELLECT-1 Technical Reportcited by 20
  15. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 7
  16. 2023SLowcal-SGD: Slow Query Points Improve Local-SGD for Stochastic Convex Optimizationcited by 4
  17. 2023Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentumcited by 3

Built on by

Not cited yet.