# Text-driven Prompt Generation for Vision-Language Models in Federated Learning

arXiv: https://arxiv.org/abs/2310.06123 · HTML: https://mlsanity.com/paper/2310.06123/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [Visual-Language Prompt Tuning with Knowledge-guided Context Optimization](https://mlsanity.com/paper/2303.13283/) (2023)
- [Unified Vision and Language Prompt Learning](https://mlsanity.com/paper/2210.07225/) (2022)
- [PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model](https://mlsanity.com/paper/2208.11625/) (2022)
- [FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning](https://mlsanity.com/paper/2302.13485/) (2023)
- [LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models](https://mlsanity.com/paper/2210.01115/) (2022)
- [Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation](https://mlsanity.com/paper/2308.15367/) (2023)
- [Efficient Multimodal Fusion via Interactive Prompting](https://mlsanity.com/paper/2304.06306/) (2023)
- [LoMar: A Local Defense Against Poisoning Attack on Federated Learning](https://mlsanity.com/paper/2201.02873/) (2022)
- [Federated Prompt Learning for Weather Foundation Models on Devices](https://mlsanity.com/paper/2305.14244/) (2023)
- [FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximation](https://mlsanity.com/paper/2112.11989/) (2021)
- [On the Convergence of Multi-Server Federated Learning with Overlapping Area](https://mlsanity.com/paper/2208.07893/) (2022)
- [Continual Adaptation of Vision Transformers for Federated Learning](https://mlsanity.com/paper/2306.09970/) (2023)
- [Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging](https://mlsanity.com/paper/2306.01176/) (2023)

## Built on by (most cited first, 17 of 17)

- [Harmonizing Generalization and Personalization in Federated Prompt Learning](https://mlsanity.com/paper/2405.09771/) (2024)
- [Federated Large Language Models: Current Progress and Future Directions](https://mlsanity.com/paper/2409.15723/) (2024)
- [FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models](https://mlsanity.com/paper/2407.19953/) (2024)
- [FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning](https://mlsanity.com/paper/2404.15182/) (2024)
- [Federated In-Context Learning: Iterative Refinement for Improved Answer Quality](https://mlsanity.com/paper/2506.07440/) (2025)
- [Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models](https://mlsanity.com/paper/2602.19926/) (2026)
- [Open-Vocabulary Federated Learning with Multimodal Prototyping](https://mlsanity.com/paper/2404.01232/) (2024)
- [pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models](https://mlsanity.com/paper/2507.05394/) (2025)
- [VLLFL: A Vision-Language Model Based Lightweight Federated Learning Framework for Smart Agriculture](https://mlsanity.com/paper/2504.13365/) (2025)
- [Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion](https://mlsanity.com/paper/2506.21144/) (2025)
- [Federated Domain Generalization with Domain-specific Soft Prompts Generation](https://mlsanity.com/paper/2509.20807/) (2025)
- [Leveraging Foundation Models for Efficient Federated Learning in Resource-restricted Edge Networks](https://mlsanity.com/paper/2409.09273/) (2024)
- [FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2506.16218/) (2025)
- [SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks](https://mlsanity.com/paper/2608.10144/) (2026)
- [FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning](https://mlsanity.com/paper/2503.22263/) (2025)
- [Cooperative Pseudo Labeling for Unsupervised Federated Classification](https://mlsanity.com/paper/2510.10100/) (2025)
- [PFedMA: Who Should Join Next as Federated Multi-Agent Orchestration Evolves?](https://mlsanity.com/paper/2610.19458/) (2026)

## Abstract (arXiv)

Prompt learning for vision-language models, e.g., CoOp, has shown great success in adapting CLIP to different downstream tasks, making it a promising solution for federated learning due to computational reasons. Existing prompt learning techniques replace hand-crafted text prompts with learned vectors that offer improvements on seen classes, but struggle to generalize to unseen classes. Our work addresses this challenge by proposing Federated Text-driven Prompt Generation (FedTPG), which learns a unified prompt generation network across multiple remote clients in a scalable manner. The prompt generation network is conditioned on task-related text input, thus is context-aware, making it suitable to generalize for both seen and unseen classes. Our comprehensive empirical evaluations on nine diverse image classification datasets show that our method is superior to existing federated prompt learning methods, that achieve overall better generalization on both seen and unseen classes and is also generalizable to unseen datasets.