# Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning

arXiv: https://arxiv.org/abs/2306.01669 · HTML: https://mlsanity.com/paper/2306.01669/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 43 of its 51 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Dash: Semi-Supervised Learning with Dynamic Thresholding](https://mlsanity.com/paper/2109.00650/) (2021)
- [Unified Vision and Language Prompt Learning](https://mlsanity.com/paper/2210.07225/) (2022)
- [ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](https://mlsanity.com/paper/2204.08790/) (2022)
- [Domain Adaptation via Prompt Learning](https://mlsanity.com/paper/2202.06687/) (2022)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)

## Built on by (most cited first, 30 of 30)

- [PromptKD: Unsupervised Prompt Distillation for Vision-Language Models](https://mlsanity.com/paper/2403.02781/) (2024)
- [Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation](https://mlsanity.com/paper/2402.18334/) (2024)
- [Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data](https://mlsanity.com/paper/2406.10502/) (2024)
- [Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning](https://mlsanity.com/paper/2405.11756/) (2024)
- [Cascade Prompt Learning for Vision-Language Model Adaptation](https://mlsanity.com/paper/2409.17805/) (2024)
- [PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts](https://mlsanity.com/paper/2308.01313/) (2023)
- [Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification](https://mlsanity.com/paper/2311.07593/) (2023)
- [Logits DeConfusion with CLIP for Few-Shot Learning](https://mlsanity.com/paper/2504.12104/) (2025)
- [Tuning Vision-Language Models with Candidate Labels by Prompt Alignment](https://mlsanity.com/paper/2407.07638/) (2024)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [Revisiting semi-supervised learning in the era of foundation models](https://mlsanity.com/paper/2503.09707/) (2025)
- [VLM-CPL: Consensus Pseudo Labels from Vision-Language Models for Annotation-Free Pathological Image Classification](https://mlsanity.com/paper/2403.15836/) (2024)
- [Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Models](https://mlsanity.com/paper/2505.13317/) (2025)
- [Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class](https://mlsanity.com/paper/2404.16717/) (2024)
- [FATE: A Prompt-Tuning-Based Semi-Supervised Learning Framework for Extremely Limited Labeled Data](https://mlsanity.com/paper/2504.09828/) (2025)
- [Triple Expert Learning from Noisy Labels for Semi-Supervised Vision Foundation Model Adaptation](https://mlsanity.com/paper/2608.09052/) (2026)
- [Learning from True-False Labels via Multi-modal Prompt Retrieving](https://mlsanity.com/paper/2405.15228/) (2024)
- [Enhancing Semi-supervised Learning with Zero-shot Pseudolabels](https://mlsanity.com/paper/2502.12584/) (2025)
- [Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization](https://mlsanity.com/paper/2505.07675/) (2025)
- [Transferable Model-agnostic Vision-Language Model Adaptation for Efficient Weak-to-Strong Generalization](https://mlsanity.com/paper/2508.08604/) (2025)
- [SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation](https://mlsanity.com/paper/2501.14148/) (2025)
- [Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning](https://mlsanity.com/paper/2504.11930/) (2025)
- [Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin](https://mlsanity.com/paper/2505.02056/) (2025)
- [PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models](https://mlsanity.com/paper/2506.00910/) (2025)
- [Cooperative Pseudo Labeling for Unsupervised Federated Classification](https://mlsanity.com/paper/2510.10100/) (2025)
- [Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models](https://mlsanity.com/paper/2510.20477/) (2025)
- [Human-Corrected Labels Learning: Enhancing Labels Quality via Human Correction of VLMs Discrepancies](https://mlsanity.com/paper/2511.09063/) (2025)
- [Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective](https://mlsanity.com/paper/2512.10244/) (2025)
- [SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2604.27596/) (2026)
- [Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition](https://mlsanity.com/paper/2607.19889/) (2026)

## Abstract (arXiv)

Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a "second generation" of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that (1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at https://github.com/BatsResearch/menghini-neurips23-code.