mlsanityData to 7 Oct 2026

Improving Zero-Shot Generalization for CLIP with Synthesized Prompts

arXiv 2307.07397 ↗First posted 14 Jul 2023cs.CV cs.LGcited by 51

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

With the growing interest in pretrained vision-language models like CLIP, recent research has focused on adapting these models to downstream tasks. Despite achieving promising results, most existing methods require labeled data for all classes, which may not hold in real-world applications due to the long tail and Zipf's law. For example, some classes may lack labeled data entirely, such as emerging concepts. To address this problem, we propose a plug-and-play generative approach called SyntHesIzed Prompts~(SHIP) to improve existing fine-tuning methods. Specifically, we follow variational autoencoders to introduce a generator that reconstructs the visual features by inputting the synthesized prompts and the corresponding class names to the textual encoder of CLIP. In this manner, we easily obtain the synthesized features for the remaining label-only classes. Thereafter, we fine-tune CLIP with off-the-shelf methods by combining labeled and synthesized features. Extensive experiments on base-to-new generalization, cross-dataset transfer learning, and generalized zero-shot learning demonstrate the superiority of our approach. The code is available at urlhttps://github.com/mrflogs/SHIP.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 55 references that are in the map. First posted on arXiv 14 Jul 2023.

Builds on

38 of 55 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Learning to Decompose Visual Features with Latent Textual Prompts9 Oct
  2. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Models3 Oct
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  4. 2022Prompt Distribution Learning6 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2013Describing Textures in the Wildcited by 2,110
19 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  8. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  9. 2019f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learningcited by 164
  10. 2020Attribute Prototype Network for Zero-Shot Learningcited by 160
  11. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  12. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  13. 2021Counterfactual Zero-Shot and Open-Set Visual Recognitioncited by 95
  14. 2018Multi-modal Cycle-consistent Generalized Zero-Shot Learningcited by 87
  15. 2019Leveraging the Invariant Side of Generative Zero-Shot Learningcited by 83
  16. 2019Transferable Contrastive Network for Generalized Zero-Shot Learningcited by 64
  17. 2021HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learningcited by 57
  18. 2020Invertible Zero-Shot Recognition Flowscited by 48
  19. 2022MSDN: Mutually Semantic Distillation Network for Zero-Shot Learningcited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 26; 2025: 14; 2026: 9.

51 papers.

Sort:

By area:

  1. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  2. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  3. 2023CLIP in Medical Imaging: A Surveycited by 40
  4. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  5. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  6. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillationcited by 18
  7. 2024LoRA-IR: Taming Low-Rank Experts for Efficient All-in-One Image Restorationcited by 18
  8. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  9. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  10. 2024Progressive Semantic-Guided Vision Transformer for Zero-Shot Learningcited by 17
  11. 2024Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learningcited by 16
  12. 2024Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidancecited by 14
38 more
  1. 2024DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detectioncited by 14
  2. 2023Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Modelscited by 13
  3. 2024Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learningcited by 12
  4. 2024SAFT: Towards Out-of-Distribution Generalization in Fine-Tuningcited by 12
  5. 2024Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detectioncited by 11
  6. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  7. 2024ZeroDiff: Solidified Visual-Semantic Correlation in Zero-Shot Learningcited by 10
  8. 2024Efficient Test-Time Prompt Tuning for Vision-Language Modelscited by 9
  9. 2024Data-Free Generalized Zero-Shot Learningcited by 8
  10. 2024Multi-Modal Adapter for Vision-Language Modelscited by 8
  11. 2024CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detectioncited by 6
  12. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  13. 2025GenZSL: Generative Zero-Shot Learning Via Inductive Variational Autoencodercited by 5
  14. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIPcited by 4
  15. 2025Captured by Captions: On Memorization and its Mitigation in CLIP Modelscited by 4
  16. 2024Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Modelcited by 2
  17. 2024Multi-Task Domain Adaptation for Language Grounding with 3D Objectscited by 2
  18. 2024ZeroMamba: Exploring Visual State Space Model for Zero-Shot Learningcited by 2
  19. 2025Modeling Variants of Prompts for Vision-Language Modelscited by 2
  20. 2025HVI-CIDNet+: Beyond Extreme Darkness for Low-Light Image Enhancementcited by 2
  21. 2025AME: Aligned Manifold Entropy for Robust Vision-Language Distillationcited by 2
  22. 2025Causality-guided Prompt Learning for Vision-language Models via Visual Granulationcited by 2
  23. 2026Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximationcited by 2
  24. 2024Attend and Enrich: Enhanced Visual Prompt for Zero-Shot Learningcited by 1
  25. 2024Expanding Event Modality Applications through a Robust CLIP-Based Encodercited by 1
  26. 2025Enhancing Zero-Shot Image Recognition in Vision-Language Models through Human-like Concept Guidancecited by 1
  27. 2025Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruptioncited by 1
  28. 2025Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalizationcited by 1
  29. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuningcited by 0
  30. 2024Improved Feature Generating Framework for Transductive Zero-shot Learningcited by 0
  31. 2025Federated Cross-Modal Style-Aware Prompt Generationcited by 0
  32. 2026Learning Adaptive and Visually Aligned Conditions for Generative Zero-Shot Learningcited by 0
  33. 2026Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognitioncited by 0
  34. 2026Mutually Causal Semantic Distillation Network for Zero-Shot Learningcited by 0
  35. 2026Incentivizing Generative Zero-Shot Learning via Outcome-Reward Reinforcement Learning with Visual Cuescited by 0
  36. 2026CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillationcited by 0
  37. 2026DINO Eats CLIP: Adapting Beyond Knowns for Open-set 3D Object Retrievalcited by 0
  38. 2026Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Modelscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SYNC-CLIP: Synthetic Data Make CLIP Generalize Better in Data-Limited Scenarios
  2. 2025Generalizable Prompt Learning of CLIP: A Brief Overview
  3. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
  4. 2024Enhancing Vision-Language Models Generalization via Diversity-Driven Novel Feature Synthesis
  5. 2024Generalizable Prompt Tuning for Vision-Language Models
  6. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning
  7. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts
  8. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners