# Multimodal Generalized Category Discovery

arXiv: https://arxiv.org/abs/2409.11624 · HTML: https://mlsanity.com/paper/2409.11624/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [Automatically Discovering and Learning New Visual Categories with Ranking Statistics](https://mlsanity.com/paper/2002.05714/) (2020)
- [Supervised Multimodal Bitransformers for Classifying Images and Text](https://mlsanity.com/paper/1909.02950/) (2019)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [Parametric Classification for Generalized Category Discovery: A Baseline Study](https://mlsanity.com/paper/2211.11727/) (2022)
- [PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery](https://mlsanity.com/paper/2212.05590/) (2022)
- [Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data](https://mlsanity.com/paper/2104.12673/) (2021)
- [SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning](https://mlsanity.com/paper/2403.13684/) (2024)
- [N24News: A New Dataset for Multimodal News Classification](https://mlsanity.com/paper/2108.13327/) (2021)
- [Generalized Category Discovery with Decoupled Prototypical Network](https://mlsanity.com/paper/2211.15115/) (2022)
- [Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discovery](https://mlsanity.com/paper/2403.07369/) (2024)
- [Federated Generalized Category Discovery](https://mlsanity.com/paper/2305.14107/) (2023)
- [GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery](https://mlsanity.com/paper/2403.09974/) (2024)
- [CLIP-GCD: Simple Language Guided Generalized Category Discovery](https://mlsanity.com/paper/2305.10420/) (2023)
- [CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification](https://mlsanity.com/paper/2112.03562/) (2021)
- [Effective Prompt Pool Learning for Continual Category Discovery](https://mlsanity.com/paper/2407.19001/) (2024)
- [UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning](https://mlsanity.com/paper/2305.09299/) (2023)
- [Neural Language Modeling with Visual Features](https://mlsanity.com/paper/1903.02930/) (2019)

## Built on by (most cited first, 7 of 7)

- [Category Discovery: An Open-World Perspective](https://mlsanity.com/paper/2509.22542/) (2025)
- [Open Ad-hoc Categorization with Contextualized Feature Learning](https://mlsanity.com/paper/2512.16202/) (2025)
- [Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts](https://mlsanity.com/paper/2509.26227/) (2025)
- [Video-based Generalized Category Discovery via Memory-Guided Consistency-Aware Contrastive Learning](https://mlsanity.com/paper/2509.06306/) (2025)
- [Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervision](https://mlsanity.com/paper/2509.12771/) (2025)
- [In-Context Clustering with Large Language Models](https://mlsanity.com/paper/2510.08466/) (2025)
- [Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery](https://mlsanity.com/paper/2606.21446/) (2026)

## Abstract (arXiv)

Generalized Category Discovery (GCD) aims to classify inputs into both known and novel categories, a task crucial for open-world scientific discoveries. However, current GCD methods are limited to unimodal data, overlooking the inherently multimodal nature of most real-world data. In this work, we extend GCD to a multimodal setting, where inputs from different modalities provide richer and complementary information. Through theoretical analysis and empirical validation, we identify that the key challenge in multimodal GCD lies in effectively aligning heterogeneous information across modalities. To address this, we propose MM-GCD, a novel framework that aligns both the feature and output spaces of different modalities using contrastive learning and distillation techniques. MM-GCD achieves new state-of-the-art performance on the UPMC-Food101 and N24News datasets, surpassing previous methods by 11.5% and 4.7%, respectively.