# Overcoming the Communication-Performance Tradeoff in LLM Pretraining

arXiv: https://arxiv.org/abs/2508.15706 · HTML: https://mlsanity.com/paper/2508.15706/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Builds on (most cited first, 30 of 30)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Federated Optimization: Distributed Machine Learning for On-Device Intelligence](https://mlsanity.com/paper/1610.02527/) (2016)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization](https://mlsanity.com/paper/1909.13014/) (2019)
- [Error Feedback Fixes SignSGD and other Gradient Compression Schemes](https://mlsanity.com/paper/1901.09847/) (2019)
- [Don't Use Large Mini-Batches, Use Local SGD](https://mlsanity.com/paper/1808.07217/) (2018)
- [Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations](https://mlsanity.com/paper/1906.02367/) (2019)
- [FetchSGD: Communication-Efficient Federated Learning with Sketching](https://mlsanity.com/paper/2007.07682/) (2020)
- [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://mlsanity.com/paper/2312.06795/) (2023)
- [SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum](https://mlsanity.com/paper/1910.00643/) (2019)
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://mlsanity.com/paper/2311.08105/) (2023)
- [Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be](https://mlsanity.com/paper/2304.13960/) (2023)
- [Stochastic Controlled Averaging for Federated Learning with Communication Compression](https://mlsanity.com/paper/2308.08165/) (2023)
- [Understanding Top-k Sparsification in Distributed Deep Learning](https://mlsanity.com/paper/1911.08772/) (2019)
- [Momentum Provably Improves Error Feedback!](https://mlsanity.com/paper/2305.15155/) (2023)
- [Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo](https://mlsanity.com/paper/2503.09799/) (2025)
- [Near-Optimal Sparse Allreduce for Distributed Deep Learning](https://mlsanity.com/paper/2201.07598/) (2022)
- [INTELLECT-1 Technical Report](https://mlsanity.com/paper/2412.01152/) (2024)
- [Trade-offs of Local SGD at Scale: An Empirical Study](https://mlsanity.com/paper/2110.08133/) (2021)
- [LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch](https://mlsanity.com/paper/2501.07124/) (2025)
- [Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory](https://mlsanity.com/paper/2201.02664/) (2022)
- [Convergence of Distributed Adaptive Optimization with Local Updates](https://mlsanity.com/paper/2409.13155/) (2024)
- [WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average](https://mlsanity.com/paper/2405.17517/) (2024)
- [Incentivizing Permissionless Distributed Learning of LLMs](https://mlsanity.com/paper/2505.21684/) (2025)
- [Meta-learning Optimizers for Communication-Efficient Learning](https://mlsanity.com/paper/2312.02204/) (2023)

## Built on by (most cited first, 12 of 12)

- [Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL](https://mlsanity.com/paper/2602.03839/) (2026)
- [Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet](https://mlsanity.com/paper/2603.08163/) (2026)
- [Decoupled DiLoCo for Resilient Distributed Pre-training](https://mlsanity.com/paper/2604.21428/) (2026)
- [Heterogeneous Low-Bandwidth Pre-Training of LLMs](https://mlsanity.com/paper/2601.02360/) (2026)
- [LoRDO: Distributed Low-Rank Optimization with Infrequent Communication](https://mlsanity.com/paper/2602.04396/) (2026)
- [Learned Subspace Compression for Communication-Efficient Pipeline Parallelism](https://mlsanity.com/paper/2606.05484/) (2026)
- [Unifying Local Communications and Local Updates for LLM Pretraining](https://mlsanity.com/paper/2606.11081/) (2026)
- [Can Model Merging Improve Aggregation in DiLoCo?](https://mlsanity.com/paper/2607.03011/) (2026)
- [ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism](https://mlsanity.com/paper/2604.11947/) (2026)
- [FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs](https://mlsanity.com/paper/2606.19025/) (2026)
- [Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure](https://mlsanity.com/paper/2607.02544/) (2026)
- [Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization](https://mlsanity.com/paper/2610.11502/) (2026)

## Abstract (arXiv)

Communication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable interest due to their benefits for training large language models (LLMs) in bandwidth-constrained settings, such as across datacenters and over the internet. While these local-update methods achieve communication reduction through reduced synchronization, they still require communicating dense model-sized pseudo-gradients, resulting in a communication bottleneck even for cross-datacenter links. While quantization is often applied to reduce the pseudo-gradient's size, in the context of LLM pre-training, existing approaches have not been able to leverage sparsification without incurring significant performance degradation. In this work, we introduce SparseLoCo, a communication-efficient training algorithm for LLMs that can effectively leverage Top-k sparsification and 2-bit quantization to reach extreme sparsity in the communicated pseudo-gradient, as high as 97-99%, while achieving lower final loss than dense DiLoCo. In our empirical study of language model pre-training, we demonstrate that SparseLoCo's improvements over DiLoCo in performance and pseudo-gradient compression are maintained across dense model scales (178M-2B), an MoE transformer (645M-A273M), increasing the number of workers, and increasing communication intervals.