mlsanityData to 14 Oct 2026

FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models

arXiv 2506.16218 ↗First posted 19 Jun 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Federated prompt learning (FPL) for vision-language models is a powerful approach to collaboratively adapt models across distributed clients while preserving data privacy. However, existing FPL approaches suffer from a trade-off between performance and robustness, particularly in out-of-distribution (OOD) shifts, limiting their reliability in real-world scenarios. The inherent in-distribution (ID) data heterogeneity among different clients makes it more challenging to maintain this trade-off. To fill this gap, we introduce a Federated OOD-aware Context Optimization (FOCoOp) framework, which captures diverse distributions among clients using ID global prompts, local prompts, and OOD prompts. Specifically, FOCoOp leverages three sets of prompts to create both class-level and distribution-level separations, which adapt to OOD shifts through bi-level distributionally robust optimization. Additionally, FOCoOp improves the discrimination consistency among clients, i.e., calibrating global prompts, seemingly OOD prompts, and OOD prompts by semi-unbalanced optimal transport. The extensive experiments on real-world datasets demonstrate that FOCoOp effectively captures decentralized heterogeneous distributions and enhances robustness of different OOD shifts. The project is available at GitHub.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 83 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

50 of 83 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method29 Sep
  2. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models12 Jul
  3. 2024GalLoP: Learning Global and Local Prompts for Vision-Language Models1 Jul
  4. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning16 May
  5. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Models29 Mar
  6. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learning9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  10. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  11. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  12. 2021Model-Contrastive Federated Learningcited by 766
31 more
  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  2. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  3. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 287
  4. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  5. 2015TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Trackingcited by 165
  6. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  7. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  8. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  9. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  10. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  11. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  12. 2018Benchmarking Neural Network Robustness to Common Corruptions and Surface Variationscited by 99
  13. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  14. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  15. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  16. 2023Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detectioncited by 57
  17. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  18. 2023ID-like Prompt Learning for Few-Shot Out-of-Distribution Detectioncited by 42
  19. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 42
  20. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  21. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  22. 2022A Unified Wasserstein Distributional Robustness Framework for Adversarial Trainingcited by 36
  23. 2023Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?cited by 35
  24. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  25. 2022SphereFed: Hyperspherical Federated Learningcited by 19
  26. 2024FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detectioncited by 14
  27. 2024Rethinking the Representation in Federated Unsupervised Learning with Non-IID Datacited by 13
  28. 2024Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspectivecited by 9
  29. 2025Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Datacited by 9
  30. 2023HyperFed: Hyperbolic Prototypes Exploration with Consistent Aggregation for Non-IID Data in Federated Learningcited by 8
  31. 2022Exploiting Personalized Invariance for Better Out-of-distribution Generalization in Federated Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Cooperative Pseudo Labeling for Unsupervised Federated Classificationcited by 0
  2. 2026CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model
  2. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning
  3. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
  4. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
  5. 2026SDFed: Bridging Local Global Discrepancy via Subspace Refinement and Divergence Control in Federated Prompt Learning
  6. 2025FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models
  7. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
  8. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models