mlsanityData to 14 Oct 2026

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression

arXiv 2410.12707 ↗First posted 16 Oct 2024cs.DC cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system designed and implemented for training DNNs using geo-distributed GPUs across different computing clusters or individual devices. Decentralized training faces significant challenges regarding system design and efficiency, including: 1) the need for remote automatic differentiation (RAD), 2) support for flexible model definitions and heterogeneous software, 3) heterogeneous hardware leading to low resource utilization or the straggler problem, and 4) slow network communication. To address these challenges, in the system design, we represent the model as a directed acyclic graph of operators (OP-DAG). Each node in the DAG represents the operator in the DNNs, while the edge represents the data dependency between operators. Based on this design, 1) users are allowed to customize any DNN without caring low-level operator implementation; 2) we enable the task scheduling with the more fine-grained sub-tasks, offering more optimization space; 3) a DAG runtime executor can implement RAD withour requiring the consistent low-level ML framework versions. To enhance system efficiency, we implement a workload estimator and design an OP-Fence scheduler to cluster devices with similar bandwidths together and partition the DAG to increase throughput. Additionally, we propose an AdaTopK compressor to adaptively compress intermediate activations and gradients at the slowest communication links. To evaluate the convergence and efficiency of our system and algorithms, we train ResNet-101 and GPT-2 on three real-world testbeds using 48 GPUs connected with 8 Mbps~10 Gbps networks. Experimental results demonstrate that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 38 of its 84 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

38 of 84 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  9. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  11. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  12. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
26 more
  1. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  2. 2018An Empirical Model of Large-Batch Trainingcited by 281
  3. 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
  4. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 228
  5. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  6. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 145
  7. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  8. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 99
  9. 2013Dropout Training as Adaptive Regularizationcited by 93
  10. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  11. 2016Memory-Efficient Backpropagation Through Timecited by 78
  12. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  13. 2021Distributed Deep Learning in Open Collaborationscited by 58
  14. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  15. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  16. 2018JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Executioncited by 40
  17. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  18. 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23
  19. 2022Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learningcited by 23
  20. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  21. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  22. 2024FedImpro: Measuring and Improving Client Update in Federated Learningcited by 15
  23. 2018MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithmscited by 12
  24. 2023Evaluation and Optimization of Gradient Compression for Distributed Deep Learningcited by 7
  25. 2014Parallel Graph Partitioning for Complex Networkscited by 3
  26. 2015Incentive-Compatible Experimental Designcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  2. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 41
  3. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  4. 2025One-shot Federated Learning Methods: A Practical Guidecited by 6
  5. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralizationcited by 4
  6. 2024Photon: Federated LLM Pre-Trainingcited by 0
  7. 2026ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Trainingcited by 0