# Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discovery

arXiv: https://arxiv.org/abs/2403.07369 · HTML: https://mlsanity.com/paper/2403.07369/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [Parametric Classification for Generalized Category Discovery: A Baseline Study](https://mlsanity.com/paper/2211.11727/) (2022)
- [Dynamic Conceptional Contrastive Learning for Generalized Category Discovery](https://mlsanity.com/paper/2303.17393/) (2023)
- [PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery](https://mlsanity.com/paper/2212.05590/) (2022)
- [CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets](https://mlsanity.com/paper/2302.02551/) (2023)
- [Learning Semi-supervised Gaussian Mixture Models for Generalized Category Discovery](https://mlsanity.com/paper/2305.06144/) (2023)
- [The Herbarium Challenge 2019 Dataset](https://mlsanity.com/paper/1906.05372/) (2019)
- [Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language](https://mlsanity.com/paper/2306.16410/) (2023)
- [Asymmetric Co-Teaching for Unsupervised Cross Domain Person Re-Identification](https://mlsanity.com/paper/1912.01349/) (2019)
- [Curriculum Graph Co-Teaching for Multi-Target Domain Adaptation](https://mlsanity.com/paper/2104.00808/) (2021)
- [Boosting Co-teaching with Compression Regularization for Label Noise](https://mlsanity.com/paper/2104.13766/) (2021)

## Built on by (most cited first, 30 of 31)

- [SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery](https://mlsanity.com/paper/2408.14371/) (2024)
- [GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery](https://mlsanity.com/paper/2403.09974/) (2024)
- [Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery](https://mlsanity.com/paper/2410.19213/) (2024)
- [ProtoGCD: Unified and Unbiased Prototype Learning for Generalized Category Discovery](https://mlsanity.com/paper/2504.03755/) (2025)
- [Effective Prompt Pool Learning for Continual Category Discovery](https://mlsanity.com/paper/2407.19001/) (2024)
- [MOS: Modeling Object-Scene Associations in Generalized Category Discovery](https://mlsanity.com/paper/2503.12035/) (2025)
- [Generate, Refine, and Encode: Leveraging Synthesized Novel Samples for On-the-Fly Fine-Grained Category Discovery](https://mlsanity.com/paper/2507.04051/) (2025)
- [Multimodal Generalized Category Discovery](https://mlsanity.com/paper/2409.11624/) (2024)
- [Category Discovery: An Open-World Perspective](https://mlsanity.com/paper/2509.22542/) (2025)
- [Learning Part Knowledge to Facilitate Category Understanding for Fine-Grained Generalized Category Discovery](https://mlsanity.com/paper/2503.16782/) (2025)
- [Open-Vocabulary Domain Generalization in Urban-Scene Segmentation](https://mlsanity.com/paper/2602.18853/) (2026)
- [Open Ad-hoc Categorization with Contextualized Feature Learning](https://mlsanity.com/paper/2512.16202/) (2025)
- [SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery](https://mlsanity.com/paper/2602.17395/) (2026)
- [TDEC: Deep Embedded Image Clustering with Transformer and Distribution Information](https://mlsanity.com/paper/2603.26746/) (2026)
- [PartCo: Part-Level Correspondence Priors Enhance Category Discovery](https://mlsanity.com/paper/2509.22769/) (2025)
- [Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts](https://mlsanity.com/paper/2509.26227/) (2025)
- [Relational Retrieval: Leveraging Known-Novel Interactions for Generalized Category Discovery](https://mlsanity.com/paper/2605.09420/) (2026)
- [Any Image Restoration via Efficient Spatial-Frequency Degradation Adaptation](https://mlsanity.com/paper/2504.14249/) (2025)
- [Video-based Generalized Category Discovery via Memory-Guided Consistency-Aware Contrastive Learning](https://mlsanity.com/paper/2509.06306/) (2025)
- [AR2-4FV: Anchored Referring and Re-identification for Long-Term Grounding in Fixed-View Videos](https://mlsanity.com/paper/2603.07758/) (2026)
- [Agile-VLA: Few-Shot Industrial Pose Rectification via Implicit Affordance Anchoring](https://mlsanity.com/paper/2603.22899/) (2026)
- [CNMBI: Determining the Number of Clusters Using Center Pairwise Matching and Boundary Filtering](https://mlsanity.com/paper/2603.26744/) (2026)
- [Deep Image Clustering Based on Curriculum Learning and Density Information](https://mlsanity.com/paper/2604.03306/) (2026)
- [Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervision](https://mlsanity.com/paper/2509.12771/) (2025)
- [Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery](https://mlsanity.com/paper/2602.19910/) (2026)
- [PACO: Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery](https://mlsanity.com/paper/2604.11484/) (2026)
- [The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery](https://mlsanity.com/paper/2604.14176/) (2026)
- [OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism](https://mlsanity.com/paper/2604.14762/) (2026)
- [SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2604.27596/) (2026)
- [Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery](https://mlsanity.com/paper/2606.21446/) (2026)

## Abstract (arXiv)

In this paper, we study the problem of Generalized Category Discovery (GCD), which aims to cluster unlabeled data from both known and unknown categories using the knowledge of labeled data from known categories. Current GCD methods rely on only visual cues, which however neglect the multi-modality perceptive nature of human cognitive processes in discovering novel visual categories. To address this, we propose a two-phase TextGCD framework to accomplish multi-modality GCD by exploiting powerful Visual-Language Models. TextGCD mainly includes a retrieval-based text generation (RTG) phase and a cross-modality co-teaching (CCT) phase. First, RTG constructs a visual lexicon using category tags from diverse datasets and attributes from Large Language Models, generating descriptive texts for images in a retrieval manner. Second, CCT leverages disparities between textual and visual modalities to foster mutual learning, thereby enhancing visual GCD. In addition, we design an adaptive class aligning strategy to ensure the alignment of category perceptions between modalities as well as a soft-voting mechanism to integrate multi-modality cues. Experiments on eight datasets show the large superiority of our approach over state-of-the-art methods. Notably, our approach outperforms the best competitor, by 7.7% and 10.8% in All accuracy on ImageNet-1k and CUB, respectively.