# GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery

arXiv: https://arxiv.org/abs/2403.09974 · HTML: https://mlsanity.com/paper/2403.09974/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 59 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2102.03526/) (2021)
- [A Unified Objective for Novel Class Discovery](https://mlsanity.com/paper/2108.08536/) (2021)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Automatically Discovering and Learning New Visual Categories with Ranking Statistics](https://mlsanity.com/paper/2002.05714/) (2020)
- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [Learning to cluster in order to transfer across domains and tasks](https://mlsanity.com/paper/1711.10125/) (2017)
- [Aligning Bag of Regions for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2302.13996/) (2023)

## Built on by (most cited first, 27 of 27)

- [Effective Prompt Pool Learning for Continual Category Discovery](https://mlsanity.com/paper/2407.19001/) (2024)
- [Multimodal Generalized Category Discovery](https://mlsanity.com/paper/2409.11624/) (2024)
- [Category Discovery: An Open-World Perspective](https://mlsanity.com/paper/2509.22542/) (2025)
- [Open-World Deepfake Attribution via Confidence-Aware Asymmetric Learning](https://mlsanity.com/paper/2512.12667/) (2025)
- [Learning Part Knowledge to Facilitate Category Understanding for Fine-Grained Generalized Category Discovery](https://mlsanity.com/paper/2503.16782/) (2025)
- [A Hidden Stumbling Block in Generalized Category Discovery: Distracted Attention](https://mlsanity.com/paper/2507.14315/) (2025)
- [FUSAR-KLIP: Towards Multimodal Foundation Models for Remote Sensing](https://mlsanity.com/paper/2509.23927/) (2025)
- [Open Ad-hoc Categorization with Contextualized Feature Learning](https://mlsanity.com/paper/2512.16202/) (2025)
- [SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery](https://mlsanity.com/paper/2602.17395/) (2026)
- [IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment](https://mlsanity.com/paper/2603.19862/) (2026)
- [PartCo: Part-Level Correspondence Priors Enhance Category Discovery](https://mlsanity.com/paper/2509.22769/) (2025)
- [Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts](https://mlsanity.com/paper/2509.26227/) (2025)
- [Relational Retrieval: Leveraging Known-Novel Interactions for Generalized Category Discovery](https://mlsanity.com/paper/2605.09420/) (2026)
- [The Master-Slave Encoder Model for Improving Patent Text Summarization: A New Approach to Combining Specifications and Claims](https://mlsanity.com/paper/2411.14072/) (2024)
- [Video-based Generalized Category Discovery via Memory-Guided Consistency-Aware Contrastive Learning](https://mlsanity.com/paper/2509.06306/) (2025)
- [Revisiting Continual Semantic Segmentation with Pre-trained Vision Models](https://mlsanity.com/paper/2508.04267/) (2025)
- [Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervision](https://mlsanity.com/paper/2509.12771/) (2025)
- [Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery](https://mlsanity.com/paper/2602.19910/) (2026)
- [Learning through Creation: A Hash-Free Framework for On-the-Fly Category Discovery](https://mlsanity.com/paper/2603.13858/) (2026)
- [Learning Like Humans: Analogical Concept Learning for Generalized Category Discovery](https://mlsanity.com/paper/2603.19918/) (2026)
- [PACO: Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery](https://mlsanity.com/paper/2604.11484/) (2026)
- [The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery](https://mlsanity.com/paper/2604.14176/) (2026)
- [SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2604.27596/) (2026)
- [Generalized Category Discovery under Domain Shifts: From Vision to Vision-Language Models](https://mlsanity.com/paper/2605.00906/) (2026)
- [Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery](https://mlsanity.com/paper/2606.21446/) (2026)
- [Test-Time Generalized Category Discovery](https://mlsanity.com/paper/2609.33937/) (2026)
- [Relational Semantic Grounding for Test-Time Discovery](https://mlsanity.com/paper/2610.14721/) (2026)

## Abstract (arXiv)

Given unlabelled datasets containing both old and new categories, generalized category discovery (GCD) aims to accurately discover new classes while correctly classifying old classes. Current GCD methods only use a single visual modality of information, resulting in a poor classification of visually similar classes. As a different modality, text information can provide complementary discriminative information, which motivates us to introduce it into the GCD task. However, the lack of class names for unlabelled data makes it impractical to utilize text information. To tackle this challenging problem, in this paper, we propose a Text Embedding Synthesizer (TES) to generate pseudo text embeddings for unlabelled samples. Specifically, our TES leverages the property that CLIP can generate aligned vision-language features, converting visual embeddings into tokens of the CLIP's text encoder to generate pseudo text embeddings. Besides, we employ a dual-branch framework, through the joint learning and instance consistency of different modality branches, visual and semantic information mutually enhance each other, promoting the interaction and fusion of visual and text knowledge. Our method unlocks the multi-modal potentials of CLIP and outperforms the baseline methods by a large margin on all GCD benchmarks, achieving new state-of-the-art. Our code is available at: https://github.com/enguangW/GET.