# Subclass Distillation

arXiv: https://arxiv.org/abs/2002.03936 · HTML: https://mlsanity.com/paper/2002.03936/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 23 of 23)

- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)
- [Transfer without Forgetting](https://mlsanity.com/paper/2206.00388/) (2022)
- [Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information](https://mlsanity.com/paper/2401.08732/) (2024)
- [Supervision Complexity and its Role in Knowledge Distillation](https://mlsanity.com/paper/2301.12245/) (2023)
- [Weighted Distillation with Unlabeled Examples](https://mlsanity.com/paper/2210.06711/) (2022)
- [How to Train the Teacher Model for Effective Knowledge Distillation](https://mlsanity.com/paper/2407.18041/) (2024)
- [SLaM: Student-Label Mixing for Distillation with Unlabeled Examples](https://mlsanity.com/paper/2302.03806/) (2023)
- [On the Efficiency of Subclass Knowledge Distillation in Classification Tasks](https://mlsanity.com/paper/2109.05587/) (2021)
- [Towards the Fundamental Limits of Knowledge Transfer over Finite Domains](https://mlsanity.com/paper/2310.07838/) (2023)
- [MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding](https://mlsanity.com/paper/2101.01881/) (2021)
- [Subclass Knowledge Distillation with Known Subclass Labels](https://mlsanity.com/paper/2207.08063/) (2022)
- [The Fixed Sub-Center: A Better Way to Capture Data Complexity](https://mlsanity.com/paper/2203.12928/) (2022)
- [Layerwise Bregman Representation Learning with Applications to Knowledge Distillation](https://mlsanity.com/paper/2209.07080/) (2022)
- [Emotion-Oriented Behavior Model Using Deep Learning](https://mlsanity.com/paper/2311.14674/) (2023)
- [Linear Projections of Teacher Embeddings for Few-Class Distillation](https://mlsanity.com/paper/2409.20449/) (2024)
- [Learning from Weakly-labeled Web Videos via Exploring Sub-Concepts](https://mlsanity.com/paper/2101.03713/) (2021)
- [Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search](https://mlsanity.com/paper/2104.00698/) (2021)
- [Self-Supervised Knowledge Transfer via Loosely Supervised Auxiliary Tasks](https://mlsanity.com/paper/2110.12696/) (2021)
- [On information captured by neural networks: connections with memorization and generalization](https://mlsanity.com/paper/2306.15918/) (2023)
- [Heterogeneous Federated Learning Using Knowledge Codistillation](https://mlsanity.com/paper/2310.02549/) (2023)
- [Probing clustering in neural network representations](https://mlsanity.com/paper/2311.07864/) (2023)
- [Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity](https://mlsanity.com/paper/2407.04291/) (2024)
- [A Shared Valence Axis Across Modern LLMs and Human EEG: The Saturation Regularity](https://mlsanity.com/paper/2606.00129/) (2026)

## Abstract (arXiv)

After a large "teacher" neural network has been trained on labeled data, the probabilities that the teacher assigns to incorrect classes reveal a lot of information about the way in which the teacher generalizes. By training a small "student" model to match these probabilities, it is possible to transfer most of the generalization ability of the teacher to the student, often producing a much better small model than directly training the student on the training data. The transfer works best when there are many possible classes because more is then revealed about the function learned by the teacher, but in cases where there are only a few possible classes we show that we can improve the transfer by forcing the teacher to divide each class into many subclasses that it invents during the supervised training. The student is then trained to match the subclass probabilities. For datasets where there are known, natural subclasses we demonstrate that the teacher learns similar subclasses and these improve distillation. For clickthrough datasets where the subclasses are unknown we demonstrate that subclass distillation allows the student to learn faster and better.