# Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery

arXiv: https://arxiv.org/abs/2410.19213 · HTML: https://mlsanity.com/paper/2410.19213/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Neural Prototype Trees for Interpretable Fine-grained Image Recognition](https://mlsanity.com/paper/2012.02046/) (2020)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [A Unified Objective for Novel Class Discovery](https://mlsanity.com/paper/2108.08536/) (2021)
- [HashNet: Deep Learning to Hash by Continuation](https://mlsanity.com/paper/1702.00758/) (2017)
- [Neighborhood Contrastive Learning for Novel Class Discovery](https://mlsanity.com/paper/2106.10731/) (2021)
- [Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes](https://mlsanity.com/paper/2111.15000/) (2021)
- [AutoNovel: Automatically Discovering and Learning Novel Visual Categories](https://mlsanity.com/paper/2106.15252/) (2021)
- [Parametric Classification for Generalized Category Discovery: A Baseline Study](https://mlsanity.com/paper/2211.11727/) (2022)
- [Dynamic Conceptional Contrastive Learning for Generalized Category Discovery](https://mlsanity.com/paper/2303.17393/) (2023)
- [Central Similarity Quantization for Efficient Image and Video Retrieval](https://mlsanity.com/paper/1908.00347/) (2019)
- [ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition](https://mlsanity.com/paper/2208.10431/) (2022)
- [Feature Learning based Deep Supervised Hashing with Pairwise Labels](https://mlsanity.com/paper/1511.03855/) (2015)
- [Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data](https://mlsanity.com/paper/2104.12673/) (2021)
- [Concept-level Debugging of Part-Prototype Networks](https://mlsanity.com/paper/2205.15769/) (2022)
- [One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective](https://mlsanity.com/paper/2109.14449/) (2021)
- [Parametric Information Maximization for Generalized Category Discovery](https://mlsanity.com/paper/2212.00334/) (2022)
- [Generalized Category Discovery with Decoupled Prototypical Network](https://mlsanity.com/paper/2211.15115/) (2022)
- [Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discovery](https://mlsanity.com/paper/2403.07369/) (2024)
- [Federated Generalized Category Discovery](https://mlsanity.com/paper/2305.14107/) (2023)
- [Proto2Proto: Can you recognize the car, the way I do?](https://mlsanity.com/paper/2204.11830/) (2022)
- [Dynamic Fraud Detection: Integrating Reinforcement Learning into Graph Neural Networks](https://mlsanity.com/paper/2409.09892/) (2024)
- [Machine Learning-Based Research on the Adaptability of Adolescents to Online Education](https://mlsanity.com/paper/2408.16849/) (2024)
- [DRAL: Deep Reinforcement Adaptive Learning for Multi-UAVs Navigation in Unknown Indoor Environment](https://mlsanity.com/paper/2409.03930/) (2024)
- [Research on Improved U-net Based Remote Sensing Image Segmentation Algorithm](https://mlsanity.com/paper/2408.12672/) (2024)

## Built on by (most cited first, 21 of 21)

- [ProtoGCD: Unified and Unbiased Prototype Learning for Generalized Category Discovery](https://mlsanity.com/paper/2504.03755/) (2025)
- [Generate, Refine, and Encode: Leveraging Synthesized Novel Samples for On-the-Fly Fine-Grained Category Discovery](https://mlsanity.com/paper/2507.04051/) (2025)
- [Category Discovery: An Open-World Perspective](https://mlsanity.com/paper/2509.22542/) (2025)
- [Open-World Deepfake Attribution via Confidence-Aware Asymmetric Learning](https://mlsanity.com/paper/2512.12667/) (2025)
- [Open-Vocabulary Domain Generalization in Urban-Scene Segmentation](https://mlsanity.com/paper/2602.18853/) (2026)
- [This looks like what? Challenges and Future Research Directions for Part-Prototype Models](https://mlsanity.com/paper/2502.09340/) (2025)
- [TDEC: Deep Embedded Image Clustering with Transformer and Distribution Information](https://mlsanity.com/paper/2603.26746/) (2026)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts](https://mlsanity.com/paper/2509.26227/) (2025)
- [Relational Retrieval: Leveraging Known-Novel Interactions for Generalized Category Discovery](https://mlsanity.com/paper/2605.09420/) (2026)
- [TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery](https://mlsanity.com/paper/2603.08075/) (2026)
- [CNMBI: Determining the Number of Clusters Using Center Pairwise Matching and Boundary Filtering](https://mlsanity.com/paper/2603.26744/) (2026)
- [Deep Image Clustering Based on Curriculum Learning and Density Information](https://mlsanity.com/paper/2604.03306/) (2026)
- [Learning through Creation: A Hash-Free Framework for On-the-Fly Category Discovery](https://mlsanity.com/paper/2603.13858/) (2026)
- [PACO: Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery](https://mlsanity.com/paper/2604.11484/) (2026)
- [The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery](https://mlsanity.com/paper/2604.14176/) (2026)
- [OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism](https://mlsanity.com/paper/2604.14762/) (2026)
- [Unlocking Optical Prior: Spectrum-Guided Knowledge Transfer for SAR Generalized Category Discovery](https://mlsanity.com/paper/2604.22174/) (2026)
- [Test-Time Generalized Category Discovery](https://mlsanity.com/paper/2609.33937/) (2026)
- [Energy Ring: Stabilizing Feature Energy for Reliable Test-Time Discovery](https://mlsanity.com/paper/2610.14525/) (2026)
- [Relational Semantic Grounding for Test-Time Discovery](https://mlsanity.com/paper/2610.14721/) (2026)

## Abstract (arXiv)

In this paper, we study a practical yet challenging task, On-the-fly Category Discovery (OCD), aiming to online discover the newly-coming stream data that belong to both known and unknown classes, by leveraging only known category knowledge contained in labeled data. Previous OCD methods employ the hash-based technique to represent old/new categories by hash codes for instance-wise inference. However, directly mapping features into low-dimensional hash space not only inevitably damages the ability to distinguish classes and but also causes "high sensitivity" issue, especially for fine-grained classes, leading to inferior performance. To address these issues, we propose a novel Prototypical Hash Encoding (PHE) framework consisting of Category-aware Prototype Generation (CPG) and Discriminative Category Encoding (DCE) to mitigate the sensitivity of hash code while preserving rich discriminative information contained in high-dimension feature space, in a two-stage projection fashion. CPG enables the model to fully capture the intra-category diversity by representing each category with multiple prototypes. DCE boosts the discrimination ability of hash code with the guidance of the generated category prototypes and the constraint of minimum separation distance. By jointly optimizing CPG and DCE, we demonstrate that these two components are mutually beneficial towards an effective OCD. Extensive experiments show the significant superiority of our PHE over previous methods, e.g., obtaining an improvement of +5.3% in ALL ACC averaged on all datasets. Moreover, due to the nature of the interpretable prototypes, we visually analyze the underlying mechanism of how PHE helps group certain samples into either known or unknown categories. Code is available at https://github.com/HaiyangZheng/PHE.