# Search to Distill: Pearls are Everywhere but not the Eyes

arXiv: https://arxiv.org/abs/1911.09074 · HTML: https://mlsanity.com/paper/1911.09074/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2019.

## Built on by (most cited first, 26 of 26)

- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [NORM: Knowledge Distillation via N-to-One Representation Matching](https://mlsanity.com/paper/2305.13803/) (2023)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [DisWOT: Student Architecture Search for Distillation WithOut Training](https://mlsanity.com/paper/2303.15678/) (2023)
- [Compatibility-aware Heterogeneous Visual Search](https://mlsanity.com/paper/2105.06047/) (2021)
- [Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation](https://mlsanity.com/paper/2105.12971/) (2021)
- [Distilling a Powerful Student Model via Online Knowledge Distillation](https://mlsanity.com/paper/2103.14473/) (2021)
- [EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge](https://mlsanity.com/paper/2311.10986/) (2023)
- [Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledge](https://mlsanity.com/paper/2110.01293/) (2021)
- [LANA: Latency Aware Network Acceleration](https://mlsanity.com/paper/2107.10624/) (2021)
- [DistPro: Searching A Fast Knowledge Distillation Process via Meta Optimization](https://mlsanity.com/paper/2204.05547/) (2022)
- [FNAS: Uncertainty-Aware Fast Neural Architecture Search](https://mlsanity.com/paper/2105.11694/) (2021)
- [Improving Differentiable Architecture Search via Self-Distillation](https://mlsanity.com/paper/2302.05629/) (2023)
- [Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Models](https://mlsanity.com/paper/2303.09639/) (2023)
- [NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension](https://mlsanity.com/paper/2211.12759/) (2022)
- [AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family](https://mlsanity.com/paper/2111.03555/) (2021)
- [Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search](https://mlsanity.com/paper/2206.13130/) (2022)
- [A Novel Architecture Slimming Method for Network Pruning and Knowledge Distillation](https://mlsanity.com/paper/2202.10461/) (2022)
- [Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey](https://mlsanity.com/paper/2208.10498/) (2022)
- [Meta-prediction Model for Distillation-Aware NAS on Unseen Datasets](https://mlsanity.com/paper/2305.16948/) (2023)
- [Group channel pruning and spatial attention distilling for object detection](https://mlsanity.com/paper/2306.01526/) (2023)
- [Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation](https://mlsanity.com/paper/2502.06192/) (2025)
- [Mutually-aware Sub-Graphs Differentiable Architecture Search](https://mlsanity.com/paper/2107.04324/) (2021)
- [A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation](https://mlsanity.com/paper/2107.07331/) (2021)
- [Adaptive Teaching with Shared Classifier for Knowledge Distillation](https://mlsanity.com/paper/2406.08528/) (2024)
- [Generalizing Teacher Networks for Effective Knowledge Distillation Across Student Architectures](https://mlsanity.com/paper/2407.16040/) (2024)

## Abstract (arXiv)

Standard Knowledge Distillation (KD) approaches distill the knowledge of a cumbersome teacher model into the parameters of a student model with a pre-defined architecture. However, the knowledge of a neural network, which is represented by the network's output distribution conditioned on its input, depends not only on its parameters but also on its architecture. Hence, a more generalized approach for KD is to distill the teacher's knowledge into both the parameters and architecture of the student. To achieve this, we present a new Architecture-aware Knowledge Distillation (AKD) approach that finds student models (pearls for the teacher) that are best for distilling the given teacher model. In particular, we leverage Neural Architecture Search (NAS), equipped with our KD-guided reward, to search for the best student architectures for a given teacher. Experimental results show our proposed AKD consistently outperforms the conventional NAS plus KD approach, and achieves state-of-the-art results on the ImageNet classification task under various latency settings. Furthermore, the best AKD student architecture for the ImageNet classification task also transfers well to other tasks such as million level face recognition and ensemble learning.