# Zero-shot Generative Model Adaptation via Image-specific Prompt Learning

arXiv: https://arxiv.org/abs/2304.03119 · HTML: https://mlsanity.com/paper/2304.03119/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 66 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Builds on (most cited first, 30 of 56)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)

## Built on by (most cited first, 27 of 27)

- [Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16223/) (2023)
- [COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing](https://mlsanity.com/paper/2406.08850/) (2024)
- [Strategic Preys Make Acute Predators: Enhancing Camouflaged Object Detectors by Generating Camouflaged Objects](https://mlsanity.com/paper/2308.03166/) (2023)
- [Rank-DETR for High Quality Object Detection](https://mlsanity.com/paper/2310.08854/) (2023)
- [Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models](https://mlsanity.com/paper/2312.04410/) (2023)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators](https://mlsanity.com/paper/2408.05710/) (2024)
- [A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shot](https://mlsanity.com/paper/2307.14397/) (2023)
- [Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning](https://mlsanity.com/paper/2404.14808/) (2024)
- [A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP](https://mlsanity.com/paper/2403.04294/) (2024)
- [GRA: Detecting Oriented Objects through Group-wise Rotating and Attention](https://mlsanity.com/paper/2403.11127/) (2024)
- [AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation](https://mlsanity.com/paper/2409.00342/) (2024)
- [FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models](https://mlsanity.com/paper/2403.16379/) (2024)
- [Cut-and-Paste: Subject-Driven Video Editing with Attention Control](https://mlsanity.com/paper/2311.11697/) (2023)
- [Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Survey](https://mlsanity.com/paper/2405.20024/) (2024)
- [EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models](https://mlsanity.com/paper/2410.09802/) (2024)
- [AdaFlow: Efficient Long Video Editing via Adaptive Attention Slimming And Keyframe Selection](https://mlsanity.com/paper/2502.05433/) (2025)
- [DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cycles](https://mlsanity.com/paper/2503.03651/) (2025)
- [Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2306.04893/) (2023)
- [Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials](https://mlsanity.com/paper/2511.00833/) (2025)
- [Bridging the Projection Gap: Overcoming Projection Bias Through Parameterized Distance Learning](https://mlsanity.com/paper/2309.01390/) (2023)
- [Image and Video Quality Assessment using Prompt-Guided Latent Diffusion Models for Cross-Dataset Generalization](https://mlsanity.com/paper/2406.04654/) (2024)
- [IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance](https://mlsanity.com/paper/2509.26231/) (2025)
- [Improving Generalization of Image Captioning with Unsupervised Prompt Learning](https://mlsanity.com/paper/2308.02862/) (2023)
- [BAPE: Learning an Explicit Bayes Classifier for Long-tailed Visual Recognition](https://mlsanity.com/paper/2506.23280/) (2025)
- [Everything to the Synthetic: Diffusion-driven Test-time Adaptation via Synthetic-Domain Alignment](https://mlsanity.com/paper/2406.04295/) (2024)
- [AIR: Zero-shot Generative Model Adaptation with Iterative Refinement](https://mlsanity.com/paper/2506.10895/) (2025)

## Abstract (arXiv)

Recently, CLIP-guided image synthesis has shown appealing performance on adapting a pre-trained source-domain generator to an unseen target domain. It does not require any target-domain samples but only the textual domain labels. The training is highly efficient, e.g., a few minutes. However, existing methods still have some limitations in the quality of generated images and may suffer from the mode collapse issue. A key reason is that a fixed adaptation direction is applied for all cross-domain image pairs, which leads to identical supervision signals. To address this issue, we propose an Image-specific Prompt Learning (IPL) method, which learns specific prompt vectors for each source-domain image. This produces a more precise adaptation direction for every cross-domain image pair, endowing the target-domain generator with greatly enhanced flexibility. Qualitative and quantitative evaluations on various domains demonstrate that IPL effectively improves the quality and diversity of synthesized images and alleviates the mode collapse. Moreover, IPL is independent of the structure of the generative model, such as generative adversarial networks or diffusion models. Code is available at https://github.com/Picsart-AI-Research/IPL-Zero-Shot-Generative-Model-Adaptation.