mlsanityData to 14 Oct 2026

Visual Prompt Based Personalized Federated Learning

arXiv 2303.08678 ↗First posted 15 Mar 2023cs.LG cs.CV cs.DCcited by 16

Not on the map. Why?

Abstract (arXiv)

As a popular paradigm of distributed learning, personalized federated learning (PFL) allows personalized models to improve generalization ability and robustness by utilizing knowledge from all distributed clients. Most existing PFL algorithms tackle personalization in a model-centric way, such as personalized layer partition, model regularization, and model interpolation, which all fail to take into account the data characteristics of distributed clients. In this paper, we propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients and provides that information to the aggregation model to help with classification tasks. Specifically, in each round of pFedPT training, each client generates a local personalized prompt related to local data distribution. Then, the local model is trained on the input composed of raw data and a visual prompt to learn the distribution information contained in the prompt. During model testing, the aggregated model obtains prior knowledge of the data distributions based on the prompts, which can be seen as an adaptive fine-tuning of the aggregation model to improve model performances on different clients. Furthermore, the visual prompt can be added as an orthogonal method to implement personalization on the client for existing FL methods to boost their performance. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 48 of its 58 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

48 of 58 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  11. 2022Visual Prompt Tuningcited by 1,681
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
36 more
  1. 2021Model-Contrastive Federated Learningcited by 766
  2. 2017Federated Multi-Task Learningcited by 672
  3. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  5. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  6. 2019Federated Learning with Personalization Layerscited by 541
  7. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  8. 2021Towards Personalized Federated Learningcited by 464
  9. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  10. 2020Adaptive Personalized Federated Learningcited by 358
  11. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  12. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  13. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  14. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  15. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 223
  16. 2021Personalized Federated Learning using Hypernetworkscited by 212
  17. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  18. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  19. 2019FedHealth: A Federated Transfer Learning Framework for Wearable Healthcarecited by 182
  20. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  21. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  22. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 155
  23. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  24. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  25. 2018Adversarial Reprogramming of Neural Networkscited by 116
  26. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  27. 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
  28. 2020Federated Learning via Synthetic Datacited by 54
  29. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  30. 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
  31. 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 31
  32. 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
  33. 2021Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfercited by 23
  34. 2022FedHiSyn: A Hierarchical Synchronous Federated Learning Framework for Resource and Data Heterogeneitycited by 20
  35. 2023Fusion of Global and Local Knowledge for Personalized Federated Learningcited by 14
  36. 2023FedABC: Targeting Fair Competition in Personalized Federated Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 6; 2025: 2.

16 papers.

Sort:

By area:

  1. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  2. 2023Integration of Large Language Models and Federated Learningcited by 31
  3. 2023FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clientscited by 19
  4. 2023Federated Prompt Learning for Weather Foundation Models on Devicescited by 17
  5. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  6. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9
  7. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  8. 2024FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learningcited by 6
  9. 2025Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Modelscited by 5
  10. 2024Open-Vocabulary Federated Learning with Multimodal Prototypingcited by 4
  11. 2024FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillationcited by 3
  12. 2024Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformationcited by 2
4 more
  1. 2024Evidential Federated Learning for Skin Lesion Image Classificationcited by 2
  2. 2023PAGE: Equilibrate Personalization and Generalization in Federated Learningcited by 1
  3. 2024Convergent Differential Privacy Analysis for General Federated Learningcited by 0
  4. 2025A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendationcited by 0