# CDUL: CLIP-Driven Unsupervised Learning for Multi-Label Image Classification

arXiv: https://arxiv.org/abs/2307.16634 · HTML: https://mlsanity.com/paper/2307.16634/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Local Aggregation for Unsupervised Learning of Visual Embeddings](https://mlsanity.com/paper/1903.12355/) (2019)
- [Learning a Deep ConvNet for Multi-label Classification with Partial Labels](https://mlsanity.com/paper/1902.09720/) (2019)
- [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition](https://mlsanity.com/paper/1908.07325/) (2019)
- [Presence-Only Geographical Priors for Fine-Grained Image Classification](https://mlsanity.com/paper/1906.05272/) (2019)
- [Unsupervised Person Re-identification via Multi-label Classification](https://mlsanity.com/paper/2004.09228/) (2020)
- [DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2206.09541/) (2022)
- [Multi-Label Learning from Single Positive Labels](https://mlsanity.com/paper/2106.09708/) (2021)
- [Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition](https://mlsanity.com/paper/2009.09450/) (2020)
- [Part-based Pseudo Label Refinement for Unsupervised Person Re-identification](https://mlsanity.com/paper/2203.14675/) (2022)
- [Refining Pseudo Labels with Clustering Consensus over Generations for Unsupervised Object Re-identification](https://mlsanity.com/paper/2106.06133/) (2021)
- [Structured Semantic Transfer for Multi-Label Recognition with Partial Labels](https://mlsanity.com/paper/2112.10941/) (2021)
- [Large Loss Matters in Weakly Supervised Multi-Label Classification](https://mlsanity.com/paper/2206.03740/) (2022)
- [Implicit Sample Extension for Unsupervised Person Re-Identification](https://mlsanity.com/paper/2204.06892/) (2022)
- [Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2203.02172/) (2022)

## Built on by (most cited first, 30 of 39)

- [Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory](https://mlsanity.com/paper/2403.12445/) (2024)
- [Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving](https://mlsanity.com/paper/2404.04804/) (2024)
- [Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion](https://mlsanity.com/paper/2312.10692/) (2023)
- [Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack](https://mlsanity.com/paper/2411.02669/) (2024)
- [A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP](https://mlsanity.com/paper/2403.04294/) (2024)
- [Multi-label Cluster Discrimination for Visual Representation Learning](https://mlsanity.com/paper/2407.17331/) (2024)
- [Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessment](https://mlsanity.com/paper/2404.15163/) (2024)
- [Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection](https://mlsanity.com/paper/2405.16038/) (2024)
- [Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning](https://mlsanity.com/paper/2403.01209/) (2024)
- [DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2505.11676/) (2025)
- [ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements](https://mlsanity.com/paper/2411.12044/) (2024)
- [Improving Multi-label Recognition using Class Co-Occurrence Probabilities](https://mlsanity.com/paper/2404.16193/) (2024)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [You Only Speak Once to See](https://mlsanity.com/paper/2409.18372/) (2024)
- [CLIP-SLA: Parameter-Efficient CLIP Adaptation for Continuous Sign Language Recognition](https://mlsanity.com/paper/2504.01666/) (2025)
- [Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere](https://mlsanity.com/paper/2505.11029/) (2025)
- [Vision-Language Model Guided Image Restoration](https://mlsanity.com/paper/2512.17292/) (2025)
- [Task2Box: Box Embeddings for Modeling Asymmetric Task Relationships](https://mlsanity.com/paper/2403.17173/) (2024)
- [Query-Based Knowledge Sharing for Open-Vocabulary Multi-Label Classification](https://mlsanity.com/paper/2401.01181/) (2024)
- [Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning](https://mlsanity.com/paper/2407.14904/) (2024)
- [SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition](https://mlsanity.com/paper/2407.20920/) (2024)
- [SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models](https://mlsanity.com/paper/2502.16911/) (2025)
- [Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification](https://mlsanity.com/paper/2503.16873/) (2025)
- [Adversarial Semantic and Label Perturbation Attack for Pedestrian Attribute Recognition](https://mlsanity.com/paper/2505.23313/) (2025)
- [Dual-View Alignment Learning with Hierarchical-Prompt for Class-Imbalance Multi-Label Classification](https://mlsanity.com/paper/2509.17747/) (2025)
- [Rethinking Prompting Strategies for Multi-Label Recognition with Partial Annotations](https://mlsanity.com/paper/2409.08381/) (2024)
- [Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentation](https://mlsanity.com/paper/2506.13925/) (2025)
- [Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inference](https://mlsanity.com/paper/2508.14280/) (2025)
- [More Reliable Pseudo-labels, Better Performance: A Generalized Approach to Single Positive Multi-label Learning](https://mlsanity.com/paper/2508.20381/) (2025)
- [Reproducibility Study of CDUL: CLIP-Driven Unsupervised Learning for Multi-Label Image Classification](https://mlsanity.com/paper/2405.11574/) (2024)

## Abstract (arXiv)

This paper presents a CLIP-based unsupervised learning method for annotation-free multi-label image classification, including three stages: initialization, training, and inference. At the initialization stage, we take full advantage of the powerful CLIP model and propose a novel approach to extend CLIP for multi-label predictions based on global-local image-text similarity aggregation. To be more specific, we split each image into snippets and leverage CLIP to generate the similarity vector for the whole image (global) as well as each snippet (local). Then a similarity aggregator is introduced to leverage the global and local similarity vectors. Using the aggregated similarity scores as the initial pseudo labels at the training stage, we propose an optimization framework to train the parameters of the classification network and refine pseudo labels for unobserved labels. During inference, only the classification network is used to predict the labels of the input image. Extensive experiments show that our method outperforms state-of-the-art unsupervised methods on MS-COCO, PASCAL VOC 2007, PASCAL VOC 2012, and NUS datasets and even achieves comparable results to weakly supervised classification methods.