mlsanityData to 14 Oct 2026

Unity is Power: Semi-Asynchronous Collaborative Training of Large-Scale Models with Structured Pruning in Resource-Limited Clients

arXiv 2410.08457 ↗First posted 11 Oct 2024cs.DC cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In this work, we study to release the potential of massive heterogeneous weak computing power to collaboratively train large-scale models on dispersed datasets. In order to improve both efficiency and accuracy in resource-adaptive collaborative learning, we take the first step to consider the textitunstructured pruning, textitvarying submodel architectures, textitknowledge loss, and textitstraggler challenges simultaneously. We propose a novel semi-asynchronous collaborative training framework, namely Co-S2P, with data distribution-aware structured pruning and cross-block knowledge transfer mechanism to address the above concerns. Furthermore, we provide theoretical proof that Co-S2P can achieve asymptotic optimal convergence rate of O(1/√N^*EQ). Finally, we conduct extensive experiments on two types of tasks with a real-world hardware testbed including diverse IoT devices.The experimental results demonstrate that Co-S2P improves accuracy by up to 8.8% and resource utilization by up to 1.2× compared to state-of-the-art methods, while reducing memory consumption by approximately 22% and training time by about 24% on all resource-limited devices.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 54 of its 74 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

54 of 74 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  11. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  12. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 490
38 more
  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  2. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  3. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  4. 2018Model compression via distillation and quantizationcited by 289
  5. 2019Asynchronous Federated Optimizationcited by 267
  6. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  7. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  8. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 207
  9. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  10. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  11. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  12. 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 149
  13. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  14. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  15. 2021Asynchronous Federated Learning on Heterogeneous Devices: A Surveycited by 93
  16. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 80
  17. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  18. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  19. 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
  20. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  21. 2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 38
  22. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  23. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 37
  24. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  25. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  26. 2023FedASMU: Efficient Asynchronous Federated Learning with Dynamic Staleness-aware Model Updatecited by 32
  27. 2019A Double Residual Compression Algorithm for Efficient Distributed Learningcited by 31
  28. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  29. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  30. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  31. 2020Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noisecited by 21
  32. 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 19
  33. 2021AsySQN: Faster Vertical Federated Learning Algorithms with Better Computation Resource Utilizationcited by 16
  34. 2022Federated Learning of Large Models at the Edge via Principal Sub-Model Trainingcited by 13
  35. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
  36. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
  37. 2024Thinking Forward: Memory-Efficient Federated Finetuning of Language Modelscited by 7
  38. 2023Federated Learning via Input-Output Collaborative Distillationcited by 6

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025TAMO: Fine-Grained Root Cause Analysis via Tool-Assisted LLM Agent with Multi-Modality Observation Data in Cloud-Native Systemscited by 13
  2. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  3. 2025LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devicescited by 0