# OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism

arXiv: https://arxiv.org/abs/2604.14762 · HTML: https://mlsanity.com/paper/2604.14762/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 75 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustment](https://mlsanity.com/paper/2306.15667/) (2023)
- [Parametric Classification for Generalized Category Discovery: A Baseline Study](https://mlsanity.com/paper/2211.11727/) (2022)
- [RelPose++: Recovering 6D Poses from Sparse-view Observations](https://mlsanity.com/paper/2305.04926/) (2023)
- [Learning Semi-supervised Gaussian Mixture Models for Generalized Category Discovery](https://mlsanity.com/paper/2305.06144/) (2023)
- [No Representation Rules Them All in Category Discovery](https://mlsanity.com/paper/2311.17055/) (2023)
- [The Herbarium Challenge 2019 Dataset](https://mlsanity.com/paper/1906.05372/) (2019)
- [SparsePose: Sparse-View Camera Pose Regression and Refinement](https://mlsanity.com/paper/2211.16991/) (2022)
- [MetaGCD: Learning to Continually Learn in Generalized Category Discovery](https://mlsanity.com/paper/2308.11063/) (2023)
- [Parametric Information Maximization for Generalized Category Discovery](https://mlsanity.com/paper/2212.00334/) (2022)
- [Contrastive Mean-Shift Learning for Generalized Category Discovery](https://mlsanity.com/paper/2404.09451/) (2024)
- [SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery](https://mlsanity.com/paper/2408.14371/) (2024)

## Abstract (arXiv)

Generalized Category Discovery (GCD) challenges methods to identify known and novel classes using partially labeled data, mirroring human category learning. Unlike prior GCD methods, which operate within a single modality and require dataset-specific fine-tuning, we propose a modality-agnostic GCD approach inspired by the human brain's abstract category formation. Our OmniGCD leverages modality-specific encoders (e.g., vision, audio, text, remote sensing) to process inputs, followed by dimension reduction to construct a GCD latent space, which is transformed at test-time into a representation better suited for clustering using a novel synthetically trained Transformer-based model. To evaluate OmniGCD, we introduce a zero-shot GCD setting where no dataset-specific fine-tuning is allowed, enabling modality-agnostic category discovery. Trained once on synthetic data, OmniGCD performs zero-shot GCD across 16 datasets spanning four modalities, improving classification accuracy for known and novel classes over baselines (average percentage point improvement of +6.2, +17.9, +1.5 and +12.7 for vision, text, audio and remote sensing). This highlights the importance of strong encoders while decoupling representation learning from category discovery. Improving modality-agnostic methods will propagate across modalities, enabling encoder development independent of GCD. Our work serves as a benchmark for future modality-agnostic GCD works, paving the way for scalable, human-inspired category discovery. All code is available hrefhttps://github.com/Jordan-HS/OmniGCDhere