# CLIP-driven Zero-shot Learning with Ambiguous Labels

arXiv: https://arxiv.org/abs/2603.05053 · HTML: https://mlsanity.com/paper/2603.05053/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [Semantics Disentangling for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2101.07978/) (2021)
- [TransZero: Attribute-guided Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2112.01683/) (2021)
- [Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learning](https://mlsanity.com/paper/1904.10056/) (2019)
- [GM-PLL: Graph Matching based Partial Label Learning](https://mlsanity.com/paper/1901.03073/) (2019)
- [Boosting Zero-shot Learning via Contrastive Optimization of Attribute Representations](https://mlsanity.com/paper/2207.03824/) (2022)
- [A Deep Model for Partial Multi-Label Image Classification with Curriculum Based Disambiguation](https://mlsanity.com/paper/2207.02410/) (2022)

## Built on by (most cited first, 1 of 1)

- [SMETA-ZSL:Semantic Meta-Alignment for Zero-Shot Threat Classification](https://mlsanity.com/paper/2607.09936/) (2026)

## Abstract (arXiv)

Zero-shot learning (ZSL) aims to recognize unseen classes by leveraging semantic information from seen classes, but most existing methods assume accurate class labels for training instances. However, in real-world scenarios, noise and ambiguous labels can significantly reduce the performance of ZSL. To address this, we propose a new CLIP-driven partial label zero-shot learning (CLIP-PZSL) framework to handle label ambiguity. First, we use CLIP to extract instance and label features. Then, a semantic mining block fuses these features to extract discriminative label embeddings. We also introduce a partial zero-shot loss, which assigns weights to candidate labels based on their relevance to the instance and aligns instance and label embeddings to minimize semantic mismatch. As the training goes on, the ground-truth labels are progressively identified, and the refined labels and label embeddings in turn help improve the semantic alignment of instance and label features. Comprehensive experiments on several datasets demonstrate the advantage of CLIP-PZSL.