mlsanityData to 14 Oct 2026

Multimodal Generalized Category Discovery

arXiv 2409.11624 ↗First posted 18 Sep 2024cs.CV cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Generalized Category Discovery (GCD) aims to classify inputs into both known and novel categories, a task crucial for open-world scientific discoveries. However, current GCD methods are limited to unimodal data, overlooking the inherently multimodal nature of most real-world data. In this work, we extend GCD to a multimodal setting, where inputs from different modalities provide richer and complementary information. Through theoretical analysis and empirical validation, we identify that the key challenge in multimodal GCD lies in effectively aligning heterogeneous information across modalities. To address this, we propose MM-GCD, a novel framework that aligns both the feature and output spaces of different modalities using contrastive learning and distillation techniques. MM-GCD achieves new state-of-the-art performance on the UPMC-Food101 and N24News datasets, surpassing previous methods by 11.5% and 4.7%, respectively.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

22 of 31 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  5. 2022Generalized Category Discoverycited by 211
  6. 2020Automatically Discovering and Learning New Visual Categories with Ranking Statisticscited by 143
  7. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  8. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  9. 2022Parametric Classification for Generalized Category Discovery: A Baseline Studycited by 113
  10. 2022PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discoverycited by 87
  11. 2021Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Datacited by 55
  12. 2024SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuningcited by 51
10 more
  1. 2021N24News: A New Dataset for Multimodal News Classificationcited by 40
  2. 2022Generalized Category Discovery with Decoupled Prototypical Networkcited by 35
  3. 2024Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discoverycited by 31
  4. 2023Federated Generalized Category Discoverycited by 27
  5. 2024GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discoverycited by 27
  6. 2023CLIP-GCD: Simple Language Guided Generalized Category Discoverycited by 25
  7. 2021CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classificationcited by 16
  8. 2024Effective Prompt Pool Learning for Continual Category Discoverycited by 14
  9. 2023UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learningcited by 13
  10. 2019Neural Language Modeling with Visual Featurescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025Category Discovery: An Open-World Perspectivecited by 5
  2. 2025Open Ad-hoc Categorization with Contextualized Feature Learningcited by 3
  3. 2025Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Expertscited by 2
  4. 2025Video-based Generalized Category Discovery via Memory-Guided Consistency-Aware Contrastive Learningcited by 1
  5. 2025Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervisioncited by 0
  6. 2025In-Context Clustering with Large Language Modelscited by 0
  7. 2026Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discoverycited by 0