mlsanityData to 14 Oct 2026

An Empirical Study of Federated Prompt Learning for Vision Language Model

arXiv 2505.23024 ↗First posted 29 May 2025cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The Vision Language Model (VLM) excels in aligning vision and language representations, and prompt learning has emerged as a key technique for adapting such models to downstream tasks. However, the application of prompt learning with VLM in federated learning (FL) scenarios remains underexplored. This paper systematically investigates the behavioral differences between language prompt learning (LPT) and vision prompt learning (VPT) under data heterogeneity challenges, including label skew and domain shift. We conduct extensive experiments to evaluate the impact of various FL and prompt configurations, such as client scale, aggregation strategies, and prompt length, to assess the robustness of Federated Prompt Learning (FPL). Furthermore, we explore strategies for enhancing prompt learning in complex scenarios where label skew and domain shift coexist, including leveraging both prompt types when computational resources allow. Our findings offer practical insights into optimizing prompt learning in federated settings, contributing to the broader deployment of VLMs in privacy-preserving environments.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

35 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Active Prompt Learning in Vision Language Models18 Nov
  2. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2022Visual Prompt Tuningcited by 1,681
  9. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  10. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  11. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  12. 2021Model-Contrastive Federated Learningcited by 766
20 more
  1. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  2. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  3. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  4. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  5. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  6. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  7. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  8. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  9. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  10. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  11. 2022Federated Learning on Non-IID Graphs via Structural Knowledge Sharingcited by 88
  12. 2019Accelerating Federated Learning via Momentum Gradient Descentcited by 76
  13. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  14. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 67
  15. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  16. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 46
  17. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  18. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 40
  19. 2022Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learningcited by 29
  20. 2023Exploring the Benefits of Visual Prompting in Differential Privacycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025HLFormer: Enhancing Partially Relevant Video Retrieval with Hyperbolic Learningcited by 17
  2. 2026SCOPE: Semantic Coreset with Orthogonal Projection Embeddings for Federated learningcited by 0
  3. 2026Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning
  2. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
  3. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
  4. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning
  5. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models
  6. 2023Learning to Prompt Your Domain for Vision-Language Models
  7. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learning
  8. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models