mlsanityData to 14 Oct 2026

Non-Federated Multi-Task Split Learning for Heterogeneous Sources

arXiv 2406.00150 ↗First posted 31 May 2024cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

With the development of edge networks and mobile computing, the need to serve heterogeneous data sources at the network edge requires the design of new distributed machine learning mechanisms. As a prevalent approach, Federated Learning (FL) employs parameter-sharing and gradient-averaging between clients and a server. Despite its many favorable qualities, such as convergence and data-privacy guarantees, it is well-known that classic FL fails to address the challenge of data heterogeneity and computation heterogeneity across clients. Most existing works that aim to accommodate such sources of heterogeneity stay within the FL operation paradigm, with modifications to overcome the negative effect of heterogeneous data. In this work, as an alternative paradigm, we propose a Multi-Task Split Learning (MTSL) framework, which combines the advantages of Split Learning (SL) with the flexibility of distributed network architectures. In contrast to the FL counterpart, in this paradigm, heterogeneity is not an obstacle to overcome, but a useful property to take advantage of. As such, this work aims to introduce a new architecture and methodology to perform multi-task learning for heterogeneous data sources efficiently, with the hope of encouraging the community to further explore the potential advantages we reveal. To support this promise, we first show through theoretical analysis that MTSL can achieve fast convergence by tuning the learning rate of the server and clients. Then, we compare the performance of MTSL with existing multi-task FL methods numerically on several image classification datasets to show that MTSL has advantages over FL in training speed, communication cost, and robustness to heterogeneous data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

36 of 50 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2018Federated Learning with Non-IID Datacited by 1,108
  7. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  8. 2017Federated Multi-Task Learningcited by 672
  9. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  10. 2019Federated Learning with Personalization Layerscited by 541
  11. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  12. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
24 more
  1. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  2. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  3. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  4. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  5. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  6. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
  7. 2018Distributed learning of deep neural network over multiple agentscited by 279
  8. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  9. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 223
  10. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  11. 2021Personalized Federated Learning using Hypernetworkscited by 212
  12. 2020Federated Learning of a Mixture of Global and Local Modelscited by 210
  13. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
  14. 2019Federated Evaluation of On-device Personalizationcited by 147
  15. 2019Federated Adversarial Domain Adaptationcited by 140
  16. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  17. 2019Detailed comparison of communication efficiency of split learning and federated learningcited by 77
  18. 2019Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applicationscited by 64
  19. 2020End-to-End Evaluation of Federated Learning and Split Learning for Internet of Thingscited by 64
  20. 2020Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applicationscited by 62
  21. 2020XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learningcited by 58
  22. 2020FedSL: Federated Split Learning on Distributed Sequential Data in Recurrent Neural Networkscited by 23
  23. 2021Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learningcited by 23
  24. 2021Splitfed learning without client-side synchronization: Analyzing client-side split network portion size to overall performancecited by 11

Built on by

Not cited yet.