mlsanityData to 14 Oct 2026

Heterogeneous Federated Learning with Splited Language Model

arXiv 2403.16050 ↗First posted 24 Mar 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Federated Split Learning (FSL) is a promising distributed learning paradigm in practice, which gathers the strengths of both Federated Learning (FL) and Split Learning (SL) paradigms, to ensure model privacy while diminishing the resource overhead of each client, especially on large transformer models in a resource-constrained environment, e.g., Internet of Things (IoT). However, almost all works merely investigate the performance with simple neural network models in FSL. Despite the minor efforts focusing on incorporating Vision Transformers (ViT) as model architectures, they train ViT from scratch, thereby leading to enormous training overhead in each device with limited resources. Therefore, in this paper, we harness Pre-trained Image Transformers (PITs) as the initial model, coined FedV, to accelerate the training process and improve model robustness. Furthermore, we propose FedVZ to hinder the gradient inversion attack, especially having the capability compatible with black-box scenarios, where the gradient information is unavailable. Concretely, FedVZ approximates the server gradient by utilizing a zeroth-order (ZO) optimization, which replaces the backward propagation with just one forward process. Empirically, we are the first to provide a systematic evaluation of FSL methods with PITs in real-world datasets, different partial device participations, and heterogeneous data splits. Our experiments verify the effectiveness of our algorithms.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

38 of 54 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  5. 2021Transformers in Vision: A Surveycited by 1,013
  6. 2021Model-Contrastive Federated Learningcited by 766
  7. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  8. 2020A Survey on Visual Transformercited by 654
  9. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  10. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  11. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
  12. 2018Distributed learning of deep neural network over multiple agentscited by 279
26 more
  1. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 246
  2. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  3. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  4. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  5. 2019Federated Learning for Wireless Communications: Motivation, Opportunities and Challengescited by 129
  6. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  7. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
  8. 2019Detailed comparison of communication efficiency of split learning and federated learningcited by 77
  9. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  10. 2020End-to-End Evaluation of Federated Learning and Split Learning for Internet of Thingscited by 64
  11. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  12. 2021Blockchain Assisted Decentralized Federated Learning (BLADE-FL): Performance Analysis and Resource Allocationcited by 56
  13. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 50
  14. 2020Advancements of federated learning towards privacy preservation: from federated learning to split learningcited by 34
  15. 2021Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Trainingcited by 17
  16. 2023Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answeringcited by 14
  17. 2021Addressing catastrophic forgetting for medical domain expansioncited by 12
  18. 2021Blockchain Assisted Federated Learning over Wireless Channels: Dynamic Resource Allocation and Client Schedulingcited by 12
  19. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  20. 2023Communication and Storage Efficient Federated Split Learningcited by 11
  21. 2022Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutationcited by 10
  22. 2023FeSViBS: Federated Split Learning of Vision Transformer with Block Samplingcited by 7
  23. 2023Efficient Federated Learning with Enhanced Privacy via Lottery Ticket Pruning in Edge Computingcited by 5
  24. 2023When MiniBatch SGD Meets SplitFed Learning:Convergence Analysis and Performance Evaluationcited by 4
  25. 2021Convergence Analysis of Nonconvex Distributed Stochastic Zeroth-order Coordinate Methodcited by 1
  26. 2022Zeroth-Order Stochastic Coordinate Methods for Decentralized Non-convex Optimizationcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Integration of TinyML and LargeML: A Survey of 6G and Beyondcited by 0