# Multi-fidelity Neural Architecture Search with Knowledge Distillation

arXiv: https://arxiv.org/abs/2006.08341 · HTML: https://mlsanity.com/paper/2006.08341/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 10 of 10)

- [A Survey on Computationally Efficient Neural Architecture Search](https://mlsanity.com/paper/2206.01520/) (2022)
- [SeqNAS: Neural Architecture Search for Event Sequence Classification](https://mlsanity.com/paper/2401.03246/) (2024)
- [Exploring the Intersection between Neural Architecture Search and Continual Learning](https://mlsanity.com/paper/2206.05625/) (2022)
- [Learning by Teaching, with Application to Neural Architecture Search](https://mlsanity.com/paper/2103.07009/) (2021)
- [AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family](https://mlsanity.com/paper/2111.03555/) (2021)
- [SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Search](https://mlsanity.com/paper/2402.13204/) (2024)
- [Fruit Classification System with Deep Learning and Neural Architecture Search](https://mlsanity.com/paper/2406.01869/) (2024)
- [A Continuous Encoding-Based Representation for Efficient Multi-Fidelity Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2509.01943/) (2025)
- [Use of static surrogates in hyperparameter optimization](https://mlsanity.com/paper/2103.07963/) (2021)
- [Discovered, Not Designed: Population Evolution for Collaborative and Compute-Intensive Model Discovery](https://mlsanity.com/paper/2610.05950/) (2026)

## Abstract (arXiv)

Neural architecture search (NAS) targets at finding the optimal architecture of a neural network for a problem or a family of problems. Evaluations of neural architectures are very time-consuming. One of the possible ways to mitigate this issue is to use low-fidelity evaluations, namely training on a part of a dataset, fewer epochs, with fewer channels, etc. In this paper, we propose a bayesian multi-fidelity method for neural architecture search: MF-KD. The method relies on a new approach to low-fidelity evaluations of neural architectures by training for a few epochs using a knowledge distillation. Knowledge distillation adds to a loss function a term forcing a network to mimic some teacher network. We carry out experiments on CIFAR-10, CIFAR-100, and ImageNet-16-120. We show that training for a few epochs with such a modified loss function leads to a better selection of neural architectures than training for a few epochs with a logistic loss. The proposed method outperforms several state-of-the-art baselines.