mlsanityData to 14 Oct 2026

Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning

arXiv 2310.18285 ↗First posted 27 Oct 2023cs.LG cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve state-of-the-art performance with improved efficiency in various computer vision tasks. This suggests a promising paradigm shift of adapting pre-trained ViT models to Federated Learning (FL) settings. However, the challenge of data heterogeneity among FL clients presents a significant hurdle in effectively deploying ViT models. Existing Generalized FL (GFL) and Personalized FL (PFL) methods have limitations in balancing performance across both global and local data distributions. In this paper, we present a novel algorithm, SGPT, that integrates GFL and PFL approaches by employing a unique combination of both shared and group-specific prompts. This design enables SGPT to capture both common and group-specific features. A key feature of SGPT is its prompt selection module, which facilitates the training of a single global model capable of automatically adapting to diverse local client data distributions without the need for local fine-tuning. To effectively train the prompts, we utilize block coordinate descent (BCD), learning from common feature information (shared prompts), and then more specialized knowledge (group prompts) iteratively. Theoretically, we justify that learning the proposed prompts can reduce the gap between global and local performance. Empirically, we conduct experiments on both label and feature heterogeneity settings in comparison with state-of-the-art baselines, along with extensive ablation studies, to substantiate the superior performance of SGPT.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

48 of 63 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2022Visual Prompt Tuningcited by 1,681
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  12. 2021Transformers in Vision: A Surveycited by 1,013
36 more
  1. 2021Learning to Prompt for Continual Learningcited by 944
  2. 2021Model-Contrastive Federated Learningcited by 766
  3. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  6. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  7. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  8. 2020Federated Learning with Matched Averagingcited by 564
  9. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  10. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  11. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  12. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  13. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  14. 2021Personalized Federated Learning using Hypernetworkscited by 212
  15. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  16. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
  17. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  18. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  19. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  20. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  21. 2020Survey of Personalization Techniques for Federated Learningcited by 145
  22. 2020Adversarial Continual Learningcited by 142
  23. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  24. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  25. 2021Federated Reconstruction: Partially Local Federated Learningcited by 79
  26. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  27. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  28. 2021Cluster-driven Graph Federated Learning over Multiple Domainscited by 44
  29. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  30. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  31. 2023Efficient Multimodal Fusion via Interactive Promptingcited by 32
  32. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  33. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  34. 2023Optimizing the Collaboration Structure in Cross-Silo Federated Learningcited by 19
  35. 2016A Primer on Coordinate Descent Algorithmscited by 17
  36. 2023On Fairness of Medical Image Classification with Multiple Sensitive Attributes via Learning Orthogonal Representationscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  2. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Methodcited by 22
  3. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  4. 2024DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Modelscited by 10
  5. 2025Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Modelscited by 5
  6. 2025LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learningcited by 4
  7. 2024FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillationcited by 3
  8. 2025FedPOB: Sample-Efficient Federated Prompt Optimization via Banditscited by 3
  9. 2024Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformationcited by 2
  10. 2025SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learningcited by 2
  11. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  12. 2024Modality Alignment Meets Federated Broadcastingcited by 1
8 more
  1. 2025pFedFair: Towards Optimal Group Fairness-Accuracy Trade-off in Heterogeneous Federated Learningcited by 1
  2. 2025FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Modelscited by 1
  3. 2025Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptationcited by 1
  4. 2024FedEGG: Federated Learning with Explicit Global Guidancecited by 0
  5. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learningcited by 0
  6. 2025Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformerscited by 0
  7. 2026FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentationcited by 0
  8. 2026FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Modelscited by 0