mlsanityData to 14 Oct 2026

Text-driven Prompt Generation for Vision-Language Models in Federated Learning

arXiv 2310.06123 ↗First posted 9 Oct 2023cs.CV cs.AIcited by 17

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Prompt learning for vision-language models, e.g., CoOp, has shown great success in adapting CLIP to different downstream tasks, making it a promising solution for federated learning due to computational reasons. Existing prompt learning techniques replace hand-crafted text prompts with learned vectors that offer improvements on seen classes, but struggle to generalize to unseen classes. Our work addresses this challenge by proposing Federated Text-driven Prompt Generation (FedTPG), which learns a unified prompt generation network across multiple remote clients in a scalable manner. The prompt generation network is conditioned on task-related text input, thus is context-aware, making it suitable to generalize for both seen and unseen classes. Our comprehensive empirical evaluations on nine diverse image classification datasets show that our method is superior to existing federated prompt learning methods, that achieve overall better generalization on both seen and unseen classes and is also generalizable to unseen datasets.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 37 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

28 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  2. 2022Unified Vision and Language Prompt Learning13 Oct
  3. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2013Describing Textures in the Wildcited by 2,125
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  9. 2019Natural Adversarial Examplescited by 1,408
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  11. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  12. 2021True Few-Shot Learning with Language Modelscited by 344
11 more
  1. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  2. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  3. 2022LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Modelscited by 59
  4. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  5. 2023Efficient Multimodal Fusion via Interactive Promptingcited by 32
  6. 2022LoMar: A Local Defense Against Poisoning Attack on Federated Learningcited by 31
  7. 2023Federated Prompt Learning for Weather Foundation Models on Devicescited by 17
  8. 2021FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximationcited by 10
  9. 2022On the Convergence of Multi-Server Federated Learning with Overlapping Areacited by 8
  10. 2023Continual Adaptation of Vision Transformers for Federated Learningcited by 6
  11. 2023Cooperative Hardware-Prompt Learning for Snapshot Compressive Imagingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024Harmonizing Generalization and Personalization in Federated Prompt Learningcited by 27
  2. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  3. 2024FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Modelscited by 17
  4. 2024FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learningcited by 16
  5. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 8
  6. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 5
  7. 2024Open-Vocabulary Federated Learning with Multimodal Prototypingcited by 4
  8. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 4
  9. 2025VLLFL: A Vision-Language Model Based Lightweight Federated Learning Framework for Smart Agriculturecited by 3
  10. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusioncited by 3
  11. 2025Federated Domain Generalization with Domain-specific Soft Prompts Generationcited by 3
  12. 2024Leveraging Foundation Models for Efficient Federated Learning in Resource-restricted Edge Networkscited by 2
5 more
  1. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Modelscited by 2
  2. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  3. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learningcited by 0
  4. 2025Cooperative Pseudo Labeling for Unsupervised Federated Classificationcited by 0
  5. 2026PFedMA: Who Should Join Next as Federated Multi-Agent Orchestration Evolves?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Replication Study: Federated Text-Driven Prompt Generation for Vision-Language Models
  2. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
  3. 2023Learning to Prompt Your Domain for Vision-Language Models
  4. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
  5. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
  6. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
  7. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
  8. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model