mlsanityData to 7 Oct 2026

DiLoCo: Distributed Low-Communication Training of Language Models

arXiv 2311.08105 ↗First posted 14 Nov 2023cs.LG cs.CLcited by 115

Not on the map. Why?

Abstract (arXiv)

Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightly interconnected accelerators, with devices exchanging gradients and other intermediate states at each optimization step. While it is difficult to build and maintain a single computing cluster hosting many accelerators, it might be easier to find several computing clusters each hosting a smaller number of devices. In this work, we propose a distributed optimization algorithm, Distributed Low-Communication (DiLoCo), that enables training of language models on islands of devices that are poorly connected. The approach is a variant of federated averaging, where the number of inner steps is large, the inner optimizer is AdamW, and the outer optimizer is Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8 workers performs as well as fully synchronous optimization while communicating 500 times less. DiLoCo exhibits great robustness to the data distribution of each worker. It is also robust to resources becoming unavailable over time, and vice versa, it can seamlessly leverage resources that become available during training.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

32 of 38 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2021Robust fine-tuning of zero-shot modelscited by 785
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  11. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  12. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
20 more
  1. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  2. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  3. 2022Patching open-vocabulary models by interpolating weightscited by 187
  4. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  5. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  6. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  7. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  8. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  9. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  10. 2021Learning Neural Network Subspacescited by 82
  11. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  12. 2021Distributed Deep Learning in Open Collaborationscited by 58
  13. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  14. 2022A Survey on Heterogeneous Federated Learningcited by 42
  15. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  16. 2023Why (and When) does Local SGD Generalize Better than SGD?cited by 26
  17. 2020Faster On-Device Training Using New Federated Momentum Algorithmcited by 25
  18. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
  19. 2022Revisiting adapters with adversarial trainingcited by 18
  20. 2023Git-Theta: A Git Extension for Collaborative Development of Machine Learning Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 23; 2025: 43; 2026: 48.

115 papers.

Sort:

By area:

  1. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  2. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  3. 2025Dion: Distributed Orthonormalized Updatescited by 64
  4. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  5. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  6. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 34
  7. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  8. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 27
  9. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  10. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  11. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 23
  12. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
38 more
  1. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
  2. 2025Decentralized Diffusion Modelscited by 12
  3. 2025Overcoming the Communication-Performance Tradeoff in LLM Pretrainingcited by 12
  4. 2024DeMo: Decoupled Momentum Optimizationcited by 10
  5. 2025Eager Updates For Overlapped Communication and Computation in DiLoCocited by 10
  6. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 10
  7. 2025Effective Quantization of Muon Optimizer Statescited by 10
  8. 2024DrJAX: Scalable and Differentiable MapReduce Primitives in JAXcited by 9
  9. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 7
  10. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  11. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 7
  12. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 7
  13. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
  14. 2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
  15. 2025SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradientscited by 6
  16. 2026Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internetcited by 6
  17. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 6
  18. 2024Worldwide Federated Training of Language Modelscited by 5
  19. 2024No Need to Talk: Asynchronous Mixture of Language Modelscited by 5
  20. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5
  21. 2026Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RLcited by 5
  22. 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
  23. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 4
  24. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  25. 2025HDEE: Heterogeneous Domain Expert Ensemblecited by 4
  26. 2025SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networkscited by 4
  27. 2025NoLoCo: No-all-reduce Low Communication Training Method for Large Modelscited by 4
  28. 2025DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Clustercited by 4
  29. 2025Towards a future space-based, highly scalable AI infrastructure system designcited by 4
  30. 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Trainingcited by 4
  31. 2026Decoupled DiLoCo for Resilient Distributed Pre-trainingcited by 4
  32. 2025Incentivizing Permissionless Distributed Learning of LLMscited by 3
  33. 2025DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Modelscited by 3
  34. 2025Model Parallelism With Subnetwork Data Parallelismcited by 3
  35. 2025Paris: A Decentralized Trained Open-Weight Diffusion Modelcited by 3
  36. 2023Meta-learning Optimizers for Communication-Efficient Learningcited by 2
  37. 2024FEDSTR: Money-In AI-Out | A Decentralized Marketplace for Federated Learning and LLM Training on the NOSTR Protocolcited by 2
  38. 2024Exploring Scaling Laws for Local SGD in Large Language Model Trainingcited by 2

50 most cited of 115 All 115 (JSON)