mlsanityData to 7 Oct 2026

Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models

arXiv 2410.10114 ↗First posted 14 Oct 2024cs.LG cs.CL cs.CVcited by 16

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods are habitually restricted to the traditional FL paradigm, where the participating clients are generally only allowed to download a single globally aggregated model from the server. While justifiable for training full-sized models under federated settings, in this work, we argue that this paradigm is ill-suited for lightweight prompts. By facilitating the clients to download multiple pre-aggregated prompts as fixed non-local experts, we propose Personalized Federated Mixture of Adaptive Prompts (pFedMoAP), a novel FL framework that personalizes the prompt learning process through the lens of Mixture of Experts (MoE). pFedMoAP implements a local attention-based gating network that learns to generate enhanced text features for better alignment with local image data, benefiting from both local and downloaded non-local adaptive prompt experts. Extensive experiments on 9 datasets under various federated settings demonstrate the efficacy of the proposed pFedMoAP algorithm. The code is available at https://github.com/ljaiverson/pFedMoAP.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 57 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

44 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method29 Sep
  2. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning16 May
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2019Advances and Open Problems in Federated Learningcited by 3,021
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  10. 2024Mixtral of Expertscited by 1,785
  11. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  12. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
28 more
  1. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  2. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  3. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  5. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  6. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  7. 2019Federated Learning with Personalization Layerscited by 533
  8. 2021Towards Personalized Federated Learningcited by 462
  9. 2020Adaptive Personalized Federated Learningcited by 353
  10. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  11. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  12. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  13. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  14. 2020Survey of Personalization Techniques for Federated Learningcited by 144
  15. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  16. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  17. 2021Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learningcited by 53
  18. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  19. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  20. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  21. 2021Federated Mixture of Expertscited by 27
  22. 2020PFL-MoE: Personalized Federated Learning Based on Mixture of Expertscited by 21
  23. 2023FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learningcited by 21
  24. 2024MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysiscited by 14
  25. 2022PGFed: Personalize Each Client's Global Objective for Federated Learningcited by 13
  26. 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
  27. 2023Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settingscited by 7
  28. 2023Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 2.

16 papers.

Sort:

By area:

  1. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  2. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  3. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  4. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusioncited by 3
  5. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 3
  6. 2025Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approachcited by 3
  7. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Graftingcited by 2
  8. 2025TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learningcited by 2
  9. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computingcited by 2
  10. 2026Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learningcited by 1
  11. 2025Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalizationcited by 0
  12. 2025A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendationcited by 0
4 more
  1. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0
  2. 2025Cooperative Pseudo Labeling for Unsupervised Federated Classificationcited by 0
  3. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Promptscited by 0
  4. 2026RoboFL: Federated Expert Assembly for World Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learning
  2. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning
  3. 2025Replication Study: Federated Text-Driven Prompt Generation for Vision-Language Models
  4. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
  5. 2023Learning to Prompt Your Domain for Vision-Language Models
  6. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model
  7. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
  8. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models