# Class-relation Knowledge Distillation for Novel Class Discovery

arXiv: https://arxiv.org/abs/2307.09158 · HTML: https://mlsanity.com/paper/2307.09158/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2102.03526/) (2021)
- [A Unified Objective for Novel Class Discovery](https://mlsanity.com/paper/2108.08536/) (2021)
- [Learning to cluster in order to transfer across domains and tasks](https://mlsanity.com/paper/1711.10125/) (2017)
- [Neighborhood Contrastive Learning for Novel Class Discovery](https://mlsanity.com/paper/2106.10731/) (2021)
- [AutoNovel: Automatically Discovering and Learning Novel Visual Categories](https://mlsanity.com/paper/2106.15252/) (2021)
- [Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation](https://mlsanity.com/paper/2107.03358/) (2021)
- [Multi-class Classification without Multi-class Labels](https://mlsanity.com/paper/1901.00544/) (2019)
- [OpenMix: Reviving Known Knowledge for Discovering Novel Visual Categories in An Open World](https://mlsanity.com/paper/2004.05551/) (2020)
- [Novel Class Discovery without Forgetting](https://mlsanity.com/paper/2207.10659/) (2022)
- [Meta Discovery: Learning to Discover Novel Classes given Very Limited Data](https://mlsanity.com/paper/2102.04002/) (2021)

## Built on by (most cited first, 23 of 23)

- [Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery](https://mlsanity.com/paper/2310.19776/) (2023)
- [Active Generalized Category Discovery](https://mlsanity.com/paper/2403.04272/) (2024)
- [HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shifts](https://mlsanity.com/paper/2408.04591/) (2024)
- [Novel Class Discovery for Long-tailed Recognition](https://mlsanity.com/paper/2308.02989/) (2023)
- [Happy: A Debiased Learning Framework for Continual Generalized Category Discovery](https://mlsanity.com/paper/2410.06535/) (2024)
- [Novel Class Discovery for Ultra-Fine-Grained Visual Categorization](https://mlsanity.com/paper/2405.06283/) (2024)
- [Dual-level Adaptive Self-Labeling for Novel Class Discovery in Point Cloud Segmentation](https://mlsanity.com/paper/2407.12489/) (2024)
- [AnomalyNCD: Towards Novel Anomaly Class Discovery in Industrial Scenarios](https://mlsanity.com/paper/2410.14379/) (2024)
- [Generalized Category Discovery via Reciprocal Learning and Class-Wise Distribution Regularization](https://mlsanity.com/paper/2506.02334/) (2025)
- [Category Discovery: An Open-World Perspective](https://mlsanity.com/paper/2509.22542/) (2025)
- [Learning Part Knowledge to Facilitate Category Understanding for Fine-Grained Generalized Category Discovery](https://mlsanity.com/paper/2503.16782/) (2025)
- [Generalized Category Discovery under Domain Shift: A Frequency Domain Perspective](https://mlsanity.com/paper/2511.00573/) (2025)
- [Self-Cooperation Knowledge Distillation for Novel Class Discovery](https://mlsanity.com/paper/2407.01930/) (2024)
- [Open Ad-hoc Categorization with Contextualized Feature Learning](https://mlsanity.com/paper/2512.16202/) (2025)
- [Continual Novel Class Discovery via Feature Enhancement and Adaptation](https://mlsanity.com/paper/2405.06389/) (2024)
- [Composing Novel Classes: A Concept-Driven Approach to Generalized Category Discovery](https://mlsanity.com/paper/2410.13285/) (2024)
- [Sharpness-aware Dynamic Anchor Selection for Generalized Category Discovery](https://mlsanity.com/paper/2512.12925/) (2025)
- [On The Relationship between Visual Anomaly-free and Anomalous Representations](https://mlsanity.com/paper/2410.06576/) (2024)
- [Freeze and Cluster: A Simple Baseline for Rehearsal-Free Continual Category Discovery](https://mlsanity.com/paper/2503.09106/) (2025)
- [Intra-view and Inter-view Correlation Guided Multi-view Novel Class Discovery](https://mlsanity.com/paper/2507.12029/) (2025)
- [The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery](https://mlsanity.com/paper/2604.14176/) (2026)
- [Generalized Category Discovery under Domain Shifts: From Vision to Vision-Language Models](https://mlsanity.com/paper/2605.00906/) (2026)
- [PHOEBI: An Open-World Benchmark for Multi-Label Bacterial Identification in Phase-Contrast Microscopy](https://mlsanity.com/paper/2606.22890/) (2026)

## Abstract (arXiv)

We tackle the problem of novel class discovery, which aims to learn novel classes without supervision based on labeled data from known classes. A key challenge lies in transferring the knowledge in the known-class data to the learning of novel classes. Previous methods mainly focus on building a shared representation space for knowledge transfer and often ignore modeling class relations. To address this, we introduce a class relation representation for the novel classes based on the predicted class distribution of a model trained on known classes. Empirically, we find that such class relation becomes less informative during typical discovery training. To prevent such information loss, we propose a novel knowledge distillation framework, which utilizes our class-relation representation to regularize the learning of novel classes. In addition, to enable a flexible knowledge distillation scheme for each data point in novel classes, we develop a learnable weighting function for the regularization, which adaptively promotes knowledge transfer based on the semantic similarity between the novel and known classes. To validate the effectiveness and generalization of our method, we conduct extensive experiments on multiple benchmarks, including CIFAR100, Stanford Cars, CUB, and FGVC-Aircraft datasets. Our results demonstrate that the proposed method outperforms the previous state-of-the-art methods by a significant margin on almost all benchmarks. Code is available at hrefhttps://github.com/kleinzcy/Cr-KD-NCDhere.