mlsanityData to 14 Oct 2026

Harmonizing Generalization and Personalization in Federated Prompt Learning

arXiv 2405.09771 ↗First posted 16 May 2024cs.LGcited by 27

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Federated Prompt Learning (FPL) incorporates large pre-trained Vision-Language models (VLM) into federated learning through prompt tuning. The transferable representations and remarkable generalization capacity of VLM make them highly compatible with the integration of federated learning. Addressing data heterogeneity in federated learning requires personalization, but excessive focus on it across clients could compromise the model's ability to generalize effectively. To preserve the impressive generalization capability of VLM, it is crucial to strike a balance between personalization and generalization in FPL. To tackle this challenge, we proposed Federated Prompt Learning with CLIP Generalization and low-rank Personalization (FedPGP), which employs pre-trained CLIP to provide knowledge-guidance on the global prompt for improved generalization and incorporates a low-rank adaptation term to personalize the global prompt. Further, FedPGP integrates a prompt-wise contrastive loss to achieve knowledge guidance and personalized adaptation simultaneously, enabling a harmonious balance between personalization and generalization in FPL. We conduct extensive experiments on various datasets to explore base-to-novel generalization in both category-level and domain-level scenarios with heterogeneous data, showing the superiority of FedPGP in balancing generalization and personalization.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 62 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

45 of 62 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learning9 Oct
  2. 2023Learning to Prompt Your Domain for Vision-Language Models4 Oct
  3. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models4 Nov
  4. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  9. 2013Describing Textures in the Wildcited by 2,125
  10. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  12. 2021Model-Contrastive Federated Learningcited by 766
27 more
  1. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  2. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  3. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  4. 2019Federated Learning with Personalization Layerscited by 541
  5. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  6. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  7. 2021Towards Personalized Federated Learningcited by 464
  8. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  9. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 287
  10. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 283
  11. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  12. 2021Personalized Federated Learning using Hypernetworkscited by 212
  13. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  14. 2019Federated Evaluation of On-device Personalizationcited by 147
  15. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
  16. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  17. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  18. 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 96
  19. 2021FedProc: Prototypical Contrastive Federated Learning on Non-IID datacited by 81
  20. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  21. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 46
  22. 2021Federated Learning with Domain Generalizationcited by 41
  23. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  24. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  25. 2022Knowledge-Aware Federated Active Learning with Non-IID Datacited by 16
  26. 2023Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learningcited by 7
  27. 2022Federated Fuzzy Neural Network with Evolutionary Rule Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 9.

27 papers.

Sort:

By area:

  1. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  2. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Methodcited by 22
  3. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Modelscited by 17
  4. 2025Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Modelscited by 5
  5. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 4
  6. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusioncited by 3
  7. 2025Federated Domain Generalization with Domain-specific Soft Prompts Generationcited by 3
  8. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Modelscited by 2
  9. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  10. 2024Modality Alignment Meets Federated Broadcastingcited by 1
  11. 2025CAPT: Class-Aware Prompt Tuning for Federated Long-Tailed Learning with Vision-Language Modelcited by 1
  12. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Modelscited by 1
15 more
  1. 2026Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learningcited by 1
  2. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learningcited by 0
  3. 2025Federated Cross-Modal Style-Aware Prompt Generationcited by 0
  4. 2025Zero-Shot Decentralized Federated Learningcited by 0
  5. 2025Cooperative Pseudo Labeling for Unsupervised Federated Classificationcited by 0
  6. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Promptscited by 0
  7. 2025How (Mis)calibrated is your Federated CLIP and what to do about it?cited by 0
  8. 2026SDFed: Bridging Local Global Discrepancy via Subspace Refinement and Divergence Control in Federated Prompt Learningcited by 0
  9. 2026Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learningcited by 0
  10. 2026FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Modelscited by 0
  11. 2026Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approachcited by 0
  12. 2026On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensingcited by 0
  13. 2026Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directionscited by 0
  14. 2026Beyond Domain-Level Adaptation: Margin-Oriented Semantic-Appearance Interaction Correction for Personalized Federated Vision-Language Modelscited by 0
  15. 2026BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Federated Domain Generalization via Prompt Learning and Aggregation
  2. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model
  3. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
  4. 2025Replication Study: Federated Text-Driven Prompt Generation for Vision-Language Models
  5. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
  6. 2025FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
  7. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
  8. 2025Privacy-preserving Prompt Personalization in Federated Learning for Multimodal Large Language Models