mlsanityData to 14 Oct 2026

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training

arXiv 2407.07852 ↗First posted 10 Jul 2024cs.LG cs.DCcited by 24

Not on the map. Why?

Abstract (arXiv)

OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. We demonstrate its effectiveness by training a model across two continents and three countries, while maintaining 90-95% compute utilization. Additionally, we conduct ablations studies focusing on the algorithm's compute efficiency, scalability in the number of workers and show that its gradients can be all-reduced using FP16 without any performance degradation. Furthermore, we scale OpenDiLoCo to 3x the size of the original work, demonstrating its effectiveness for billion parameter models.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 6 of its 9 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

6 of 9 references.

References · 6

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2017Decoupled Weight Decay Regularizationcited by 1,831
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2017Mixed Precision Trainingcited by 1,156
  5. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  6. 2023Efficient Parallelization Layouts for Large-Scale Distributed Model Trainingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 11.

24 papers.

Sort:

By area:

  1. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 28
  2. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 27
  3. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
  4. 2025Eager Updates For Overlapped Communication and Computation in DiLoCocited by 11
  5. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 9
  6. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  7. 2025HDEE: Heterogeneous Domain Expert Ensemblecited by 4
  8. 2025SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networkscited by 4
  9. 2025DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Clustercited by 4
  10. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  11. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  12. 2025On the Duality between Gradient Transformations and Adapterscited by 1
12 more
  1. 2025Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systemscited by 1
  2. 2025Taming Latency and Bandwidth: A Theoretical Framework and Adaptive Algorithm for Communication-Constrained Trainingcited by 1
  3. 2026High dimensional theory of two-phase optimizerscited by 1
  4. 2026Learned Subspace Compression for Communication-Efficient Pipeline Parallelismcited by 1
  5. 2026Is Decentralized AI Governable? From Regulative Policy to Constitutive Protocolcited by 0
  6. 2026Echelon: Auditable Aggregate-Only Language-Model Adaptation Across Privacy Boundariescited by 0
  7. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEscited by 0
  8. 2026Decentralised AI Training and Inference with BlockTraincited by 0
  9. 2026Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructurecited by 0
  10. 2026Communication-Efficient LLM Adaptation over Decentralized GPU Meshescited by 0
  11. 2026AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronizationcited by 0
  12. 2026Greedy Local Learning for Language Model Pretraining: Gaps and Objective Designcited by 0