# Soft Prompt Generation for Domain Generalization

arXiv: https://arxiv.org/abs/2404.19286 · HTML: https://mlsanity.com/paper/2404.19286/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 34 of its 40 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Domain Generalization by Mutual-Information Regularization with Pre-trained Models](https://mlsanity.com/paper/2203.10789/) (2022)
- [Domain Adaptation via Prompt Learning](https://mlsanity.com/paper/2202.06687/) (2022)
- [Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization](https://mlsanity.com/paper/2009.12829/) (2020)
- [PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization](https://mlsanity.com/paper/2307.15199/) (2023)
- [Prompt Vision Transformer for Domain Generalization](https://mlsanity.com/paper/2208.08914/) (2022)
- [Bayesian Prompt Learning for Image-Language Model Generalization](https://mlsanity.com/paper/2210.02390/) (2022)
- [Prompt-based Distribution Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2312.09553/) (2023)

## Built on by (most cited first, 24 of 24)

- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [Handling Out-of-Distribution Data: A Survey](https://mlsanity.com/paper/2507.21160/) (2025)
- [Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization](https://mlsanity.com/paper/2507.02288/) (2025)
- [Generalizing vision-language models to novel domains: A comprehensive survey](https://mlsanity.com/paper/2506.18504/) (2025)
- [Reasoning-Driven Multimodal LLM for Domain Generalization](https://mlsanity.com/paper/2602.23777/) (2026)
- [PromptTA: Prompt-driven Text Adapter for Source-free Domain Generalization](https://mlsanity.com/paper/2409.14163/) (2024)
- [Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentation](https://mlsanity.com/paper/2512.03508/) (2025)
- [Federated Domain Generalization with Domain-specific Soft Prompts Generation](https://mlsanity.com/paper/2509.20807/) (2025)
- [Harmonizing and Merging Source Models for CLIP-based Domain Generalization](https://mlsanity.com/paper/2506.09446/) (2025)
- [Simulate, Refocus and Ensemble: An Attention-Refocusing Scheme for Domain Generalization](https://mlsanity.com/paper/2507.12851/) (2025)
- [Attention Head Purification: A New Perspective to Harness CLIP for Domain Generalization](https://mlsanity.com/paper/2412.07226/) (2024)
- [A Language Vision Model Approach for Automated Tumor Contouring in Radiation Oncology](https://mlsanity.com/paper/2503.14933/) (2025)
- [Text-Driven Causal Representation Learning for Source-Free Domain Generalization](https://mlsanity.com/paper/2507.09961/) (2025)
- [Latent Domain Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2511.00067/) (2025)
- [Generalizing Vision-Language Models with Dedicated Prompt Guidance](https://mlsanity.com/paper/2512.02421/) (2025)
- [Dual-Path Stable Soft Prompt Generation for Domain Generalization](https://mlsanity.com/paper/2505.18770/) (2025)
- [Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection](https://mlsanity.com/paper/2508.16976/) (2025)
- [Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations](https://mlsanity.com/paper/2508.21769/) (2025)
- [OmniVaT: Single Domain Generalization for Multimodal Visual-Tactile Learning](https://mlsanity.com/paper/2601.00352/) (2026)
- [SemAlign: Language Guided Semi-supervised Domain Generalization](https://mlsanity.com/paper/2601.11724/) (2026)
- [Optimizing Soft Prompt Tuning via Structural Evolution](https://mlsanity.com/paper/2602.16500/) (2026)
- [Hierarchical Prompt Injector for Domain Generalization Segmentation](https://mlsanity.com/paper/2609.05864/) (2026)

## Abstract (arXiv)

Large pre-trained vision language models (VLMs) have shown impressive zero-shot ability on downstream tasks with manually designed prompt. To further adapt VLMs to downstream tasks, soft prompt is proposed to replace manually designed prompt, which undergoes fine-tuning based on specific domain data. Prior prompt learning methods primarily learn a fixed prompt or residuled prompt from training samples. However, the learned prompts lack diversity and ignore information about unseen domains. In this paper, we reframe the prompt learning framework from a generative perspective and propose a simple yet efficient method for the Domain Generalization (DG) task, namely Soft Prompt Generation (SPG). Specifically, SPG consists of a two-stage training phase and an inference phase. During the training phase, we introduce soft prompt label for each domain, aiming to incorporate the generative model domain knowledge. During the inference phase, the generator of the generative model is employed to obtain instance-specific soft prompts for the unseen target domain. Extensive experiments on five domain generalization benchmarks of three DG tasks demonstrate that SPG achieves state-of-the-art performance. The code is available at https://github.com/renytek13/Soft-Prompt-Generation-with-CGAN.