mlsanityData to 14 Oct 2026

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs

arXiv 2309.01172 ↗First posted 3 Sep 2023cs.DC cs.AI cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

The rapid growth of memory and computation requirements of large language models (LLMs) has outpaced the development of hardware, hindering people who lack large-scale high-end GPUs from training or deploying LLMs. However, consumer-level GPUs, which constitute a larger market share, are typically overlooked in LLM due to their weaker computing performance, smaller storage capacity, and lower communication bandwidth. Additionally, users may have privacy concerns when interacting with remote LLMs. In this paper, we envision a decentralized system unlocking the potential vast untapped consumer-level GPUs in pre-training, inference and fine-tuning of LLMs with privacy protection. However, this system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity. To address these challenges, our system design incorporates: 1) a broker with backup pool to implement dynamic join and quit of computing providers; 2) task scheduling with hardware performance to improve system efficiency; 3) abstracting ML procedures into directed acyclic graphs (DAGs) to achieve model and task universality; 4) abstracting intermediate represention and execution planes to ensure compatibility of various devices and deep learning (DL) frameworks. Our performance analysis demonstrates that 50 RTX 3080 GPUs can achieve throughputs comparable to those of 4 H100 GPUs, which are significantly more expensive.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 49 of its 86 references that are in the map. First posted on arXiv 3 Sep 2023.

Builds on

49 of 86 references.

References · 49

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2019Deep Leakage from Gradientscited by 1,021
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  7. 2018How To Backdoor Federated Learningcited by 765
  8. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  10. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  11. 2017Countering Adversarial Images using Input Transformationscited by 520
  12. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
37 more
  1. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 334
  2. 2018An Empirical Model of Large-Batch Trainingcited by 281
  3. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  4. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  5. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  6. 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
  7. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 228
  8. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  9. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 132
  10. 2021Scaling Laws for Neural Machine Translationcited by 111
  11. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
  12. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  13. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  14. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 79
  15. 2016Memory-Efficient Backpropagation Through Timecited by 78
  16. 2022Variational Model Inversion Attackscited by 75
  17. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
  18. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  19. 2021Distributed Deep Learning in Open Collaborationscited by 58
  20. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  21. 2018Predicting the Computational Cost of Deep Learning Modelscited by 49
  22. 2020FMore: An Incentive Scheme of Multi-dimensional Auction for Federated Learning in MECcited by 48
  23. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  24. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  25. 2018JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Executioncited by 40
  26. 2021A Comprehensive Survey of Incentive Mechanism for Federated Learningcited by 38
  27. 2019A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networkscited by 37
  28. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  29. 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23
  30. 2020Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusterscited by 23
  31. 2022Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learningcited by 23
  32. 2020Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selectioncited by 20
  33. 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
  34. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  35. 2018MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithmscited by 12
  36. 2022Dynamic GPU Energy Optimization for Machine Learning Training Workloadscited by 12
  37. 2023Evaluation and Optimization of Gradient Compression for Distributed Deep Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 14; 2025: 3; 2026: 1.

20 papers.

Sort:

By area:

  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  2. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  3. 2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 45
  4. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 41
  5. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  6. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  7. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  8. 2024LongGenBench: Long-context Generation Benchmarkcited by 28
  9. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  10. 2024Should We Really Edit Language Models? On the Evaluation of Edited Language Modelscited by 18
  11. 2024FedImpro: Measuring and Improving Client Update in Federated Learningcited by 15
  12. 2024FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusioncited by 11
8 more
  1. 2024FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compressioncited by 7
  2. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralizationcited by 4
  3. 2024Improving training time and GPU utilization in geo-distributed language model trainingcited by 3
  4. 2024FEDSTR: Money-In AI-Out | A Decentralized Marketplace for Federated Learning and LLM Training on the NOSTR Protocolcited by 2
  5. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  6. 2023Exploring the Robustness of Decentralized Training for Large Language Modelscited by 1
  7. 2025A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environmentscited by 1
  8. 2025Asynchronous Decentralized SGD under Non-Convexity: A Block-Coordinate Descent Frameworkcited by 0