mlsanityData to 14 Oct 2026

Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similarity

arXiv 2412.16414 ↗First posted 21 Dec 2024math.OC cs.DC cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

In recent years, as data and problem sizes have increased, distributed learning has become an essential tool for training high-performance models. However, the communication bottleneck, especially for high-dimensional data, is a challenge. Several techniques have been developed to overcome this problem. These include communication compression and implementation of local steps, which work particularly well when there is similarity of local data samples. In this paper, we study the synergy of these approaches for efficient distributed optimization. We propose the first theoretically grounded accelerated algorithms utilizing unbiased and biased compression under data similarity, leveraging variance reduction and error feedback frameworks. Our results are of record and confirmed by experiments on different average losses and datasets.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

31 of 48 references.

References · 31

  1. 2019Advances and Open Problems in Federated Learningcited by 3,046
  2. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  3. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  4. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  5. 2019A Survey on Distributed Machine Learningcited by 188
  6. 2020On Biased Compression for Distributed Learningcited by 140
  7. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
  8. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  9. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
  10. 2019Distributed Learning with Compressed Gradient Differencescited by 112
  11. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  12. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
19 more
  1. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  2. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 78
  3. 2019Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loopcited by 68
  4. 2015Communication Complexity of Distributed Convex Learning and Optimizationcited by 60
  5. 2018Distributed learning with compressed gradientscited by 55
  6. 2020Error Compensated Distributed SGD Can Be Acceleratedcited by 45
  7. 2020Statistically Preconditioned Accelerated Gradient Method for Distributed Optimizationcited by 30
  8. 2021Federated Learning of Molecular Properties with Graph Neural Networks in a Heterogeneous Settingcited by 19
  9. 2019On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyondcited by 18
  10. 2018Katyusha X: Practical Momentum Method for Stochastic Sum-of-Nonconvex Optimizationcited by 15
  11. 2021Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guaranteescited by 15
  12. 2023Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?cited by 14
  13. 2015Differentially Private Distributed Convex Optimization via Functional Perturbationcited by 11
  14. 2023Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysiscited by 11
  15. 2024LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compressioncited by 11
  16. 2019Performance Analysis and Comparison of Distributed Machine Learning Systemscited by 9
  17. 2023Two Losses Are Better Than One: Faster Optimization Using a Cheaper Proxycited by 7
  18. 2023Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalitiescited by 7
  19. 2022Modern Distributed Data-Parallel Large-Scale Pre-training Strategies For NLP modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Accelerated Distributed Optimization with Compression and Error Feedbackcited by 4
  2. 2026Accelerated Methods with Complexity Separation Under Data Similarity for Federated Learning Problemscited by 0