Exploring the Robustness of Decentralized Training for Large Language Models
Not on the map. Why?
Abstract (arXiv)
Decentralized training of large language models has emerged as an effective way to democratize this technology. However, the potential threats associated with this approach have not been carefully discussed, which would hinder the development of decentralized training infrastructures. This paper aims to initiate discussion towards this end by exploring the robustness of decentralized training from three main perspectives. First, we demonstrate the vulnerabilities inherent in decentralized training frameworks in terms of hardware, data, and models. Second, we highlight the fundamental difference between decentralized foundation model training and vanilla federated learning, where the security techniques employed in federated learning cannot be applied directly. Third, we discuss the essential components required for a robust and efficient decentralized training framework and present a case study by modeling a concrete threat model. Our objective in this vision paper is to emphasize the importance of addressing security concerns in the context of decentralized training for large language models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 68 references that are in the map. First posted on arXiv 1 Dec 2023.
Builds on
39 of 68 references.
References · 39
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
- 2019Deep Leakage from Gradientscited by 1,004
- 2016Deep Networks with Stochastic Depthcited by 973
- 2019Towards Federated Learning at Scale: System Designcited by 826
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
27 more
- 2020iDLG: Improved Deep Leakage from Gradientscited by 367
- 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
- 2020Learning from History for Byzantine Robust Optimizationcited by 137
- 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
- 2013Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifierscited by 128
- 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 125
- 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
- 2014Deep learning with Elastic Averaging SGDcited by 90
- 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
- 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
- 2023Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneitycited by 64
- 2022Byzantine Machine Learning Made Easy by Resilient Averaging of Momentumscited by 63
- 2021Distributed Deep Learning in Open Collaborationscited by 58
- 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
- 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
- 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
- 2019BRIDGE: Byzantine-resilient Decentralized Gradient Descentcited by 45
- 2016AIDE: Fast and Communication Efficient Distributed Optimizationcited by 42
- 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
- 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
- 2022Byzantine-Robust Federated Learning with Optimal Statistical Rates and Privacy Guaranteescited by 30
- 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
- 2020Privacy Preserving Distributed Machine Learning with Federated Learningcited by 13
- 2021Regularization in ResNet with Stochastic Depthcited by 7
- 2023Byzantine-Resilient Federated Learning at Edgecited by 6
- 2022Byzantine Spectral Rankingcited by 5
- 2018Byzantine Preferential Votingcited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.