mlsanityData to 14 Oct 2026

Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning

arXiv 2306.01669 ↗First posted 2 Jun 2023cs.CV cs.LGcited by 30

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a "second generation" of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that (1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at https://github.com/BatsResearch/menghini-neurips23-code.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 51 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

43 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models11 Mar
  2. 2022Multitask Vision-Language Prompt Tuning21 Nov
  3. 2022Unified Vision and Language Prompt Learning13 Oct
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  5. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  6. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learning7 Apr
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2022Visual Prompt Tuningcited by 1,681
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
24 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  2. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  3. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  4. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  5. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  6. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  7. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  8. 2019Label Propagation for Deep Semi-supervised Learningcited by 289
  9. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  10. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  11. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 255
  12. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 148
  13. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  14. 2022Domain Adaptation via Prompt Learningcited by 128
  15. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  16. 2022Visual Prompt Tuning for Test-time Domain Adaptationcited by 97
  17. 2022Debiased Learning from Naturally Imbalanced Pseudo-Labelscited by 85
  18. 2022Debiased Self-Training for Semi-Supervised Learningcited by 62
  19. 2021Domain Prompt Learning for Efficiently Adapting CLIP to Unseen Domainscited by 39
  20. 2021The Rich Get Richer: Disparate Impact of Semi-Supervised Learningcited by 28
  21. 2021Hardness Sampling for Self-Training Based Transductive Zero-Shot Learningcited by 12
  22. 2017Transductive Zero-Shot Learning with a Self-training dictionary approachcited by 7
  23. 2022An Iterative Co-Training Transductive Framework for Zero Shot Learningcited by 3
  24. 2021TAGLETS: A System for Automatic Semi-Supervised Learning with Auxiliary Datacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 15; 2026: 3.

30 papers.

Sort:

By area:

  1. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  2. 2024Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptationcited by 27
  3. 2024Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Datacited by 20
  4. 2024Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learningcited by 19
  5. 2024Cascade Prompt Learning for Vision-Language Model Adaptationcited by 19
  6. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contextscited by 17
  7. 2023Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classificationcited by 13
  8. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  9. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignmentcited by 6
  10. 2024LEMoN: Label Error Detection using Multimodal Neighborscited by 6
  11. 2025Revisiting semi-supervised learning in the era of foundation modelscited by 6
  12. 2024VLM-CPL: Consensus Pseudo Labels from Vision-Language Models for Annotation-Free Pathological Image Classificationcited by 5
18 more
  1. 2025Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Modelscited by 3
  2. 2024Embracing Diversity: Interpretable Zero-shot classification beyond one vector per classcited by 2
  3. 2025FATE: A Prompt-Tuning-Based Semi-Supervised Learning Framework for Extremely Limited Labeled Datacited by 2
  4. 2026Triple Expert Learning from Noisy Labels for Semi-Supervised Vision Foundation Model Adaptationcited by 2
  5. 2024Learning from True-False Labels via Multi-modal Prompt Retrievingcited by 1
  6. 2025Enhancing Semi-supervised Learning with Zero-shot Pseudolabelscited by 1
  7. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimizationcited by 1
  8. 2025Transferable Model-agnostic Vision-Language Model Adaptation for Efficient Weak-to-Strong Generalizationcited by 1
  9. 2025SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptationcited by 0
  10. 2025Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learningcited by 0
  11. 2025Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margincited by 0
  12. 2025PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Modelscited by 0
  13. 2025Cooperative Pseudo Labeling for Unsupervised Federated Classificationcited by 0
  14. 2025Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Modelscited by 0
  15. 2025Human-Corrected Labels Learning: Enhancing Labels Quality via Human Correction of VLMs Discrepanciescited by 0
  16. 2025Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspectivecited by 0
  17. 2026SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learningcited by 0
  18. 2026Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  2. 2025Prototype-Guided Pseudo-Labeling with Neighborhood-Aware Consistency for Unsupervised Adaptation
  3. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts
  4. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
  5. 2023Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
  6. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting
  7. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Time
  8. 2023ECO: Ensembling Context Optimization for Vision-Language Models