mlsanityData to 7 Oct 2026

FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning

arXiv 2503.22263 ↗First posted 28 Mar 2025cs.LG cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The increasing emphasis on privacy and data security has driven the adoption of federated learning, a decentralized approach to train machine learning models without sharing raw data. Prompt learning, which fine-tunes prompt embeddings of pretrained models, offers significant advantages in federated settings by reducing computational costs and communication overheads while leveraging the strong performance and generalization capabilities of vision-language models such as CLIP. This paper addresses the intersection of federated learning and prompt learning, particularly for vision-language models. In this work, we introduce a comprehensive framework, named FLIP, to evaluate federated prompt learning algorithms. FLIP assesses the performance of 8 state-of-the-art federated prompt learning methods across 4 federated learning protocols and 12 open datasets, considering 6 distinct evaluation scenarios. Our findings demonstrate that prompt learning maintains strong generalization performance in both in-distribution and out-of-distribution settings with minimal resource consumption. This work highlights the effectiveness of federated prompt learning in environments characterized by data scarcity, unseen classes, and cross-domain distributional shifts. We open-source the code for all implemented algorithms in FLIP to facilitate further research in this domain.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 72 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

53 of 72 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models23 Jan
  2. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method29 Sep
  3. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning16 May
  4. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learning9 Oct
  5. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  6. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  7. 2022Prompt Distribution Learning6 May
  8. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  9. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2022Visual Prompt Tuningcited by 1,674
32 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  5. 2019How Can We Know What Language Models Know?cited by 953
  6. 2019Towards Federated Learning at Scale: System Designcited by 826
  7. 2018LEAF: A Benchmark for Federated Settingscited by 780
  8. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  9. 2020Federated Learning with Matched Averagingcited by 564
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  11. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  12. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  13. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  14. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  15. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  16. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  17. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  18. 2021OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalizationcited by 86
  19. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  20. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  21. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  22. 2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51
  23. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  24. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  25. 2022FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learningcited by 43
  26. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  27. 2022Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learningcited by 29
  28. 2022FLAIR: Federated Learning Annotated Image Repositorycited by 22
  29. 2022The Two Dimensions of Worst-case Training and the Integrated Effect for Out-of-domain Generalizationcited by 21
  30. 2022Alleviating the Sample Selection Bias in Few-shot Learning by Removing Projection to the Centroidcited by 20
  31. 2023Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learningcited by 20
  32. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model
  2. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
  3. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
  4. 2026SDFed: Bridging Local Global Discrepancy via Subspace Refinement and Divergence Control in Federated Prompt Learning
  5. 2024Federated Domain Generalization via Prompt Learning and Aggregation
  6. 2023Learning to Prompt Your Domain for Vision-Language Models
  7. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models
  8. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts