mlsanityData to 7 Oct 2026

Unsupervised Prompt Learning for Vision-Language Models

arXiv 2204.03649 ↗First posted 7 Apr 2022cs.CVcited by 125

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 55 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

41 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  2. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
25 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2019How Can We Know What Language Models Know?cited by 953
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  12. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  13. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  14. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  15. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 315
  16. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  17. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  18. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  19. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  20. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  21. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  22. 2021Semi-Supervised Semantic Segmentation via Adaptive Equalization Learningcited by 101
  23. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  24. 2021Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognitioncited by 29
  25. 2019Lessons from Building Acoustic Models with a Million Hours of Speechcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 38; 2024: 38; 2025: 25; 2026: 8.

125 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  3. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  4. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  5. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  6. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  7. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  8. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  9. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 150
  10. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  11. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  12. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
38 more
  1. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  2. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  3. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  4. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  5. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  6. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  7. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  8. 2022LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Modelscited by 58
  9. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learningcited by 58
  10. 2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51
  11. 2022Texts as Images in Prompt Tuning for Multi-Label Image Recognitioncited by 51
  12. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  13. 2023Natural Language-Assisted Sign Language Recognitioncited by 46
  14. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  15. 2023Black Box Few-Shot Adaptation for Vision-Language modelscited by 39
  16. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?cited by 38
  17. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
  18. 2024Boosting Vision-Language Models with Transductioncited by 36
  19. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  20. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Modelscited by 34
  21. 2022SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classificationcited by 34
  22. 2023CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learningcited by 32
  23. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuningcited by 30
  24. 2022Boosting the Transferability of Adversarial Attacks with Global Momentum Initializationcited by 29
  25. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  26. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  27. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 28
  28. 2024Label Propagation for Zero-shot Classification with Vision-Language Modelscited by 28
  29. 2023Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIPcited by 27
  30. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 27
  31. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  32. 2023OpenVIS: Open-vocabulary Video Instance Segmentationcited by 25
  33. 2023DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotationscited by 22
  34. 2024AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learningcited by 22
  35. 2023Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?cited by 21
  36. 2024Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clusteringcited by 21
  37. 2024Historical Test-time Prompt Tuning for Vision Foundation Modelscited by 21
  38. 2024Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Datacited by 19

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Unified Vision and Language Prompt Learning
  2. 2026What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
  3. 2023DPL: Decoupled Prompt Learning for Vision-Language Models
  4. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  5. 2024Revisiting Prompt Pretraining of Vision-Language Models
  6. 2023Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models
  7. 2025Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
  8. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP