mlsanityData to 14 Oct 2026

Taming Latency and Bandwidth: A Theoretical Framework and Adaptive Algorithm for Communication-Constrained Training

arXiv 2507.17346 ↗First posted 23 Jul 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Regional energy caps limit the growth of any single data center used for large-scale model training. This single-center training paradigm works when model size remains manageable, but exponential growth in the model size and computational demand challenges it. A natural alternative is to distribute training across multiple data centers over wide-area networks. This pools distributed resources, but suffers from high latency and low, time-varying bandwidth, sharply reducing throughout. Employing jointly gradient compression and delayed aggregation can alleviate communication problems, but introduces a complex three-way trade-off among compression ratio, staleness (delayed synchronization steps), and convergence rate. Existing work lacks theoretical guidance and can only propose fixed strategies, insensitive to computation and communication conditions. We address this with a new theoretical tool, decomposing the joint optimization problem into a traditional process plus multiple analyzable noise terms. Our analysis yields the first convergence rate for this setting and shows that increasing staleness exponentially amplifies the detrimental effect of compression. Leveraging these insights, we propose DeCo-SGD, which dynamically selects the compression ratio and staleness based on the real-time communication and computation conditions. DeCo-SGD achieves up to 5.07× and 1.37× speed-ups over distributed SGD and static strategy in high-latency and low, varying bandwidth networks, respectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 65 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

36 of 65 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  7. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  8. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  9. 2018Sparsified SGD with Memorycited by 376
  10. 2021Explaining Neural Scaling Lawscited by 363
  11. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  12. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
24 more
  1. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  2. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  3. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 108
  4. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  5. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
  6. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  7. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  8. 2021FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learningcited by 39
  9. 2024FedAC: An Adaptive Clustered Federated Learning Framework for Heterogeneous Datacited by 25
  10. 2024OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Trainingcited by 24
  11. 2020On Communication Compression for Distributed Optimization on Heterogeneous Datacited by 19
  12. 2024CO2: Efficient Distributed Training with Full Communication-Computation Overlapcited by 17
  13. 2021Optimal Rate Adaption in Federated Learning with Compressed Communicationscited by 15
  14. 2023EControl: Fast Distributed Optimization with Compression and Error Controlcited by 15
  15. 2024Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneitycited by 13
  16. 2024FedFa: A Fully Asynchronous Training Paradigm for Federated Learningcited by 12
  17. 2021Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updatescited by 10
  18. 2022Learnings from Federated Learning in the Real worldcited by 6
  19. 2023Layered Randomized Quantization for Communication-Efficient and Privacy-Preserving Distributed Learningcited by 6
  20. 2023Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computingcited by 5
  21. 2024Accelerating Geo-distributed Machine Learning with Network-Aware Adaptive Tree and Auxiliary Routecited by 5
  22. 2025Beyond A Single AI Cluster: A Survey of Decentralized LLM Trainingcited by 4
  23. 2024Heterogeneity-Aware Cooperative Federated Edge Learning with Adaptive Computation and Communication Compressioncited by 2
  24. 2025γ-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3