mlsanityData to 14 Oct 2026

MSfusion: A Dynamic Model Splitting Approach for Resource-Constrained Machines to Collaboratively Train Larger Models

arXiv 2407.03622 ↗First posted 4 Jul 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Training large models requires a large amount of data, as well as abundant computation resources. While collaborative learning (e.g., federated learning) provides a promising paradigm to harness collective data from many participants, training large models remains a major challenge for participants with limited resources like mobile devices. We introduce MSfusion, an effective and efficient collaborative learning framework, tailored for training larger models on resourceconstraint machines through model splitting. Specifically, a double shifting model splitting scheme is designed such that in each training round, each participant is assigned a subset of model parameters to train over local data, and aggregates with sub-models of other peers on common parameters. While model splitting significantly reduces the computation and communication costs of individual participants, additional novel designs on adaptive model overlapping and contrastive loss functions help MSfusion to maintain training effectiveness, against model shift across participants. Extensive experiments on image and NLP tasks illustrate significant advantages of MSfusion in performance and efficiency for training large models, and its strong scalability: computation cost of each participant reduces significantly as the number of participants increases.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

44 of 54 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2020Supervised Contrastive Learningcited by 2,829
  12. 2023A Survey of Large Language Modelscited by 2,576
32 more
  1. 2020Knowledge Distillation: A Surveycited by 1,603
  2. 2021Model-Contrastive Federated Learningcited by 766
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  4. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  5. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  6. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  7. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  8. 2020Understanding the Behaviour of Contrastive Losscited by 398
  9. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  10. 2019On the Efficacy of Knowledge Distillationcited by 381
  11. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  12. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  13. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
  14. 2020On Biased Compression for Distributed Learningcited by 140
  15. 2019BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learningcited by 117
  16. 2021Decentralized Federated Averagingcited by 114
  17. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  18. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  19. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
  20. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  21. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  22. 2019Detailed comparison of communication efficiency of split learning and federated learningcited by 77
  23. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  24. 2020End-to-End Evaluation of Federated Learning and Split Learning for Internet of Thingscited by 64
  25. 2021LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learningcited by 64
  26. 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 35
  27. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  28. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  29. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  30. 2022FedOBD: Opportunistic Block Dropout for Efficiently Training Large-scale Neural Networks through Federated Learningcited by 17
  31. 2021Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniquescited by 9
  32. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6

Built on by

Not cited yet.