# The Solution for Language-Enhanced Image New Category Discovery

arXiv: https://arxiv.org/abs/2407.04994 · HTML: https://mlsanity.com/paper/2407.04994/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 6 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction](https://mlsanity.com/paper/2104.07650/) (2021)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2206.09541/) (2022)
- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [Pro-tuning: Unified Prompt Tuning for Vision Tasks](https://mlsanity.com/paper/2207.14381/) (2022)
- [Texts as Images in Prompt Tuning for Multi-Label Image Recognition](https://mlsanity.com/paper/2211.12739/) (2022)
- [DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2308.01890/) (2023)
- [`A-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Prompting](https://mlsanity.com/paper/2302.07994/) (2023)

## Abstract (arXiv)

Treating texts as images, combining prompts with textual labels for prompt tuning, and leveraging the alignment properties of CLIP have been successfully applied in zero-shot multi-label image recognition. Nonetheless, relying solely on textual labels to store visual information is insufficient for representing the diversity of visual objects. In this paper, we propose reversing the training process of CLIP and introducing the concept of Pseudo Visual Prompts. These prompts are initialized for each object category and pre-trained on large-scale, low-cost sentence data generated by large language models. This process mines the aligned visual information in CLIP and stores it in class-specific visual prompts. We then employ contrastive learning to transfer the stored visual information to the textual labels, enhancing their visual representation capacity. Additionally, we introduce a dual-adapter module that simultaneously leverages knowledge from the original CLIP and new learning knowledge derived from downstream datasets. Benefiting from the pseudo visual prompts, our method surpasses the state-of-the-art not only on clean annotated text data but also on pseudo text data generated by large language models.