mlsanityData to 14 Oct 2026

DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster

arXiv 2506.21263 ↗First posted 26 Jun 2025cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The distributed training of foundation models, particularly large language models (LLMs), demands a high level of communication. Consequently, it is highly dependent on a centralized cluster with fast and reliable interconnects. Can we conduct training on slow networks and thereby unleash the power of decentralized clusters when dealing with models exceeding 100 billion parameters? In this paper, we propose DiLoCoX, a low-communication large-scale decentralized cluster training framework. It combines Pipeline Parallelism with Dual Optimizer Policy, One-Step-Delay Overlap of Communication and Local Training, and an Adaptive Gradient Compression Scheme. This combination significantly improves the scale of parameters and the speed of model pre-training. We justify the benefits of one-step-delay overlap of communication and local training, as well as the adaptive gradient compression scheme, through a theoretical analysis of convergence. Empirically, we demonstrate that DiLoCoX is capable of pre-training a 107B foundation model over a 1Gbps network. Compared to vanilla AllReduce, DiLoCoX can achieve a 357x speedup in distributed training while maintaining negligible degradation in model convergence. To the best of our knowledge, this is the first decentralized training framework successfully applied to models with over 100 billion parameters.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

15 of 26 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  10. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  11. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  12. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
3 more
  1. 2022Rank Diminishing in Deep Neural Networkscited by 47
  2. 2024OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Trainingcited by 24
  3. 2022MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloudcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Decoupled DiLoCo for Resilient Distributed Pre-trainingcited by 4
  2. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  3. 2026LoRDO: Distributed Low-Rank Optimization with Infrequent Communicationcited by 1
  4. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEscited by 0