mlsanityData to 14 Oct 2026

Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts

arXiv 2002.04013 ↗First posted 10 Feb 2020cs.DC cs.LG stat.MLcited by 43

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Many recent breakthroughs in deep learning were achieved by training increasingly larger models on massive datasets. However, training such models can be prohibitively expensive. For instance, the cluster used to train GPT-3 costs over 250 million. As a result, most researchers cannot afford to train state of the art models and contribute to their development. Hypothetically, a researcher could crowdsource the training of large neural networks with thousands of regular PCs provided by volunteers. The raw computing power of a hundred thousand 2500 desktops dwarfs that of a 250M server pod, but one cannot utilize that power efficiently with conventional distributed training methods. In this work, we propose Learning@home: a novel neural network training paradigm designed to handle large amounts of poorly connected participants. We analyze the performance, reliability, and architectural constraints of this paradigm and compare it against existing distributed training techniques.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 29 of its 78 references that are in the map. First posted on arXiv 10 Feb 2020.

Builds on

29 of 78 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 848
17 more
  1. 2018How To Backdoor Federated Learningcited by 765
  2. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  3. 2019Defending Against Neural Fake Newscited by 559
  4. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  5. 2018Analyzing Federated Learning through an Adversarial Lenscited by 351
  6. 2017Large-Scale Evolution of Image Classifierscited by 348
  7. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  8. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 137
  9. 2019Large Memory Layers with Product Keyscited by 124
  10. 2019Natural Compression for Distributed Deep Learningcited by 90
  11. 2019The HSIC Bottleneck: Deep Learning without Back-Propagationcited by 77
  12. 2018Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGDcited by 68
  13. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  14. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  15. 2015Staleness-aware Async-SGD for Distributed Deep Learningcited by 43
  16. 2019Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learningcited by 25
  17. 2019Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 5; 2024: 12; 2025: 7; 2026: 8.

43 papers.

Sort:

By area:

  1. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  2. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  3. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  4. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  5. 2021Distributed Deep Learning in Open Collaborationscited by 58
  6. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  7. 2021Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitivescited by 50
  8. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  9. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  10. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 21
  11. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  12. 2021Secure Distributed Training at Scalecited by 12
31 more
  1. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 12
  2. 2024DiPaCo: Distributed Path Compositioncited by 10
  3. 2021Distributed Deep Learning Using Volunteer Computing-Like Paradigmcited by 9
  4. 2023A Language Model of Java Methods with Train/Test Deduplicationcited by 7
  5. 2024FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compressioncited by 7
  6. 2025Nesterov Method for Asynchronous Pipeline Parallel Optimizationcited by 6
  7. 2022Training Transformers Togethercited by 5
  8. 2024Dynamic Byzantine-Robust Learning: Adapting to Switching Byzantine Workerscited by 5
  9. 2024A Question-centric Multi-experts Contrastive Learning Framework for Improving the Accuracy and Interpretability of Deep Sequential Knowledge Tracing Modelscited by 5
  10. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  11. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 4
  12. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  13. 2024Protocol Learning, Decentralized Frontier Risk and the No-Off Problemcited by 3
  14. 2025When MoE Meets Blockchain: A Trustworthy Distributed Framework of Large Modelscited by 3
  15. 2024Ravnest: Decentralized Asynchronous Training on Heterogeneous Devicescited by 2
  16. 2025Towards Decentralized and Sustainable Foundation Model Training with the Edgecited by 2
  17. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  18. 2024Video Relationship Detection Using Mixture of Expertscited by 1
  19. 2024Lower Bounds and Optimal Algorithms for Non-Smooth Convex Decentralized Optimization over Time-Varying Networkscited by 1
  20. 2025All is Not Lost: LLM Recovery without Checkpointscited by 1
  21. 2026Learned Subspace Compression for Communication-Efficient Pipeline Parallelismcited by 1
  22. 2021Machines & Influence: An Information Systems Lenscited by 0
  23. 2023Direct Neural Machine Translation with Task-level Mixture of Experts modelscited by 0
  24. 2024Social Interpretable Reinforcement Learningcited by 0
  25. 2025MoLink: Distributed and Efficient Serving Framework for Large Modelscited by 0
  26. 2026DeRelayL: Sustainable Decentralized Relay Learningcited by 0
  27. 2026Unextractable Protocol Models: Collaborative Training and Inference without Weight Materializationcited by 0
  28. 2026Mixtures of Subspaces for Bandwidth Efficient Context Parallel Trainingcited by 0
  29. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEscited by 0
  30. 2026Decentralised AI Training and Inference with BlockTraincited by 0
  31. 2026GenomeOcean Anywhere: Private WebGPU Inference for Genome MoEscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
  2. 2023Cooperative Learning for Cost-Adaptive Inference
  3. 2025Decentralized Diffusion Models
  4. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Vision
  5. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoE
  6. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
  7. 2021Low Precision Decentralized Distributed Training over IID and non-IID Data
  8. 2026∞-MoE: Generalizing Mixture of Experts to Infinite Experts