# Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data

arXiv: https://arxiv.org/abs/2406.10502 · HTML: https://mlsanity.com/paper/2406.10502/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Dash: Semi-Supervised Learning with Dynamic Thresholding](https://mlsanity.com/paper/2109.00650/) (2021)
- [Domain Adaptation via Prompt Learning](https://mlsanity.com/paper/2202.06687/) (2022)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)
- [Provably Consistent Partial-Label Learning](https://mlsanity.com/paper/2007.08929/) (2020)
- [Leveraged Weighted Loss for Partial Label Learning](https://mlsanity.com/paper/2106.05731/) (2021)
- [A survey on Semi-, Self- and Unsupervised Learning for Image Classification](https://mlsanity.com/paper/2002.08721/) (2020)
- [LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections](https://mlsanity.com/paper/2305.18287/) (2023)
- [Transductive Zero-Shot Learning with Visual Structure Constraint](https://mlsanity.com/paper/1901.01570/) (2019)
- [Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning](https://mlsanity.com/paper/2306.01669/) (2023)
- [Improving Zero-Shot Models with Label Distribution Priors](https://mlsanity.com/paper/2212.00784/) (2022)

## Built on by (most cited first, 20 of 20)

- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning](https://mlsanity.com/paper/2412.20392/) (2024)
- [Tuning Vision-Language Models with Candidate Labels by Prompt Alignment](https://mlsanity.com/paper/2407.07638/) (2024)
- [Revisiting semi-supervised learning in the era of foundation models](https://mlsanity.com/paper/2503.09707/) (2025)
- [Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Models](https://mlsanity.com/paper/2505.13317/) (2025)
- [FATE: A Prompt-Tuning-Based Semi-Supervised Learning Framework for Extremely Limited Labeled Data](https://mlsanity.com/paper/2504.09828/) (2025)
- [Tuning the Right Foundation Models is What you Need for Partial Label Learning](https://mlsanity.com/paper/2506.05027/) (2025)
- [Enhancing Semi-supervised Learning with Zero-shot Pseudolabels](https://mlsanity.com/paper/2502.12584/) (2025)
- [When Test-Time Adaptation Meets Self-Supervised Models](https://mlsanity.com/paper/2506.23529/) (2025)
- [Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score](https://mlsanity.com/paper/2507.09615/) (2025)
- [SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation](https://mlsanity.com/paper/2501.14148/) (2025)
- [Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning](https://mlsanity.com/paper/2504.11930/) (2025)
- [Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin](https://mlsanity.com/paper/2505.02056/) (2025)
- [Cooperative Pseudo Labeling for Unsupervised Federated Classification](https://mlsanity.com/paper/2510.10100/) (2025)
- [Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models](https://mlsanity.com/paper/2510.20477/) (2025)
- [Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective](https://mlsanity.com/paper/2512.10244/) (2025)
- [Closing the Confusion Loop: CLIP-Guided Alignment for Source-Free Domain Adaptation](https://mlsanity.com/paper/2602.08730/) (2026)
- [ConfusionBench: An Expert-Validated Benchmark for Confusion Recognition and Localization in Educational Videos](https://mlsanity.com/paper/2603.17267/) (2026)
- [SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2604.27596/) (2026)
- [Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning](https://mlsanity.com/paper/2610.06234/) (2026)

## Abstract (arXiv)

Fine-tuning vision-language models (VLMs) with abundant unlabeled data recently has attracted increasing attention. Existing methods that resort to the pseudolabeling strategy would suffer from heavily incorrect hard pseudolabels when VLMs exhibit low zero-shot performance in downstream tasks. To alleviate this issue, we propose a Candidate Pseudolabel Learning method, termed CPL, to fine-tune VLMs with suitable candidate pseudolabels of unlabeled data in downstream tasks. The core of our method lies in the generation strategy of candidate pseudolabels, which progressively generates refined candidate pseudolabels by both intra- and inter-instance label selection, based on a confidence score matrix for all unlabeled data. This strategy can result in better performance in true label inclusion and class-balanced instance selection. In this way, we can directly apply existing loss functions to learn with generated candidate psueudolabels. Extensive experiments on nine benchmark datasets with three learning paradigms demonstrate the effectiveness of our method. Our code can be found at https://github.com/vanillaer/CPL-ICML2024.