mlsanityData to 7 Oct 2026

Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

arXiv 2409.19610 ↗First posted 29 Sep 2024cs.LG cs.CL cs.CVcited by 21

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 48 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

30 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning16 May
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  6. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  7. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  8. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  9. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 151
  10. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  11. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  12. 2022Vision Transformers provably learn spatial structurecited by 92
16 more
  1. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  2. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  3. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  4. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  5. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
  6. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  7. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  8. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  9. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  10. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  11. 2023Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learningcited by 20
  12. 2023The Benefits of Mixup for Feature Learningcited by 19
  13. 2022Knowledge-Aware Federated Active Learning with Non-IID Datacited by 15
  14. 2022Provably Learning Diverse Features in Multi-View Data with Midpoint Mixupcited by 14
  15. 2023Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learningcited by 7
  16. 2022Federated Fuzzy Neural Network with Evolutionary Rule Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 5.

21 papers.

Sort:

By area:

  1. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  2. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  3. 2025A Survey on Efficient Vision-Language Modelscited by 19
  4. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Modelscited by 16
  5. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  6. 2024NLPrompt: Noise-Label Prompt Learning for Vision-Language Modelscited by 9
  7. 2025Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learningcited by 9
  8. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 3
  9. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Modelscited by 2
  10. 2025C^2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learningcited by 2
  11. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  12. 2024Federated Vision-Language-Recommendation with Personalized Fusioncited by 1
9 more
  1. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Modelscited by 1
  2. 2026Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learningcited by 1
  3. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learningcited by 0
  4. 2025Large-Small Model Collaborative Framework for Federated Continual Learningcited by 0
  5. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Promptscited by 0
  6. 2026SDFed: Bridging Local Global Discrepancy via Subspace Refinement and Divergence Control in Federated Prompt Learningcited by 0
  7. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
  8. 2026Model Merging as Probabilistic Inference in Fine-Tuning Parameter Spacecited by 0
  9. 2026On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model
  2. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learning
  3. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
  4. 2023Learning to Prompt Your Domain for Vision-Language Models
  5. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
  6. 2025Replication Study: Federated Text-Driven Prompt Generation for Vision-Language Models
  7. 2024Federated Domain Generalization via Prompt Learning and Aggregation
  8. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models