# Search for Better Students to Learn Distilled Knowledge

arXiv: https://arxiv.org/abs/2001.11612 · HTML: https://mlsanity.com/paper/2001.11612/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2020.

## Built on by (most cited first, 11 of 11)

- [DisWOT: Student Architecture Search for Distillation WithOut Training](https://mlsanity.com/paper/2303.15678/) (2023)
- [Simple Distillation Baselines for Improving Small Self-supervised Models](https://mlsanity.com/paper/2106.11304/) (2021)
- [Learning by Teaching, with Application to Neural Architecture Search](https://mlsanity.com/paper/2103.07009/) (2021)
- [AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family](https://mlsanity.com/paper/2111.03555/) (2021)
- [Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey](https://mlsanity.com/paper/2208.10498/) (2022)
- [Meta-prediction Model for Distillation-Aware NAS on Unseen Datasets](https://mlsanity.com/paper/2305.16948/) (2023)
- [Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation](https://mlsanity.com/paper/2502.06192/) (2025)
- [Beyond Student: An Asymmetric Network for Neural Network Inheritance](https://mlsanity.com/paper/2602.09509/) (2026)
- [Conditional Generative Data-free Knowledge Distillation](https://mlsanity.com/paper/2112.15358/) (2021)
- [Explainability and Robustness of Deep Visual Classification Models](https://mlsanity.com/paper/2301.01343/) (2023)
- [Bring the Power of Diffusion Model to Defect Detection](https://mlsanity.com/paper/2408.13845/) (2024)

## Abstract (arXiv)

Knowledge Distillation, as a model compression technique, has received great attention. The knowledge of a well-performed teacher is distilled to a student with a small architecture. The architecture of the small student is often chosen to be similar to their teacher's, with fewer layers or fewer channels, or both. However, even with the same number of FLOPs or parameters, the students with different architecture can achieve different generalization ability. The configuration of a student architecture requires intensive network architecture engineering. In this work, instead of designing a good student architecture manually, we propose to search for the optimal student automatically. Based on L1-norm optimization, a subgraph from the teacher network topology graph is selected as a student, the goal of which is to minimize the KL-divergence between student's and teacher's outputs. We verify the proposal on CIFAR10 and CIFAR100 datasets. The empirical experiments show that the learned student architecture achieves better performance than ones specified manually. We also visualize and understand the architecture of the found student.