# Rank-Reliable Teacher-Guided Fitness Approximation for Expensive Evolutionary Optimization: A TinyML Architecture Search Study

arXiv: https://arxiv.org/abs/2609.30553 · HTML: https://mlsanity.com/paper/2609.30553/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks](https://mlsanity.com/paper/1802.03796/) (2018)
- [Evaluating Efficient Performance Estimators of Neural Architectures](https://mlsanity.com/paper/2008.03064/) (2020)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [An Analytical Theory of Curriculum Learning in Teacher-Student Networks](https://mlsanity.com/paper/2106.08068/) (2021)
- [Speedy Performance Estimation for Neural Architecture Search](https://mlsanity.com/paper/2006.04492/) (2020)
- [Fast Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/1910.11609/) (2019)
- [Supervision Complexity and its Role in Knowledge Distillation](https://mlsanity.com/paper/2301.12245/) (2023)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [Progressive distillation induces an implicit curriculum](https://mlsanity.com/paper/2410.05464/) (2024)
- [NACHOS: Neural Architecture Search for Hardware Constrained Early Exit Neural Networks](https://mlsanity.com/paper/2401.13330/) (2024)
- [Knowledge Distillation via Instance-level Sequence Learning](https://mlsanity.com/paper/2106.10885/) (2021)
- [Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices](https://mlsanity.com/paper/2310.07217/) (2023)
- [Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML](https://mlsanity.com/paper/2305.14109/) (2023)
- [Dynamic Ensemble of Low-fidelity Experts: Mitigating NAS "Cold-Start"](https://mlsanity.com/paper/2302.00932/) (2023)
- [Advances in Small-Footprint Keyword Spotting: A Comprehensive Review of Efficient Models and Algorithms](https://mlsanity.com/paper/2506.11169/) (2025)
- [OASI: Objective-Aware Surrogate Initialization for Multi-Objective Bayesian Optimization in TinyML Keyword Spotting](https://mlsanity.com/paper/2512.19739/) (2025)
- [SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Search](https://mlsanity.com/paper/2402.13204/) (2024)
- [Meta knowledge assisted Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2504.21545/) (2025)

## Abstract (arXiv)

Expensive evolutionary search does not always need an exact fitness estimate for every candidate. It often needs a reliable answer to a simpler question: which candidate is better? We address this need through Teacher-Guided Learning NSGA-II (TGL-NSGA-II), a low-fidelity framework for constrained Tiny Machine Learning (TinyML) neural architecture search. A pretrained teacher organizes samples into strata defined jointly by difficulty and class. Each candidate then undergoes KD-Lite, a short and capped knowledge-distillation procedure on a compact training set, before being scored on a separate stratified evaluation set. This teacher-guided score is fused with a Gaussian-process surrogate to select candidates for full evaluation. For a fixed candidate population, we analyse evaluation variance, score concentration, pairwise rank inversion, expected Kendall-τ, first-front identification, and hypervolume perturbation. We also derive a variance-aware fusion weight and a capacity-adaptive distillation rule. On keyword spotting and bird-call classification, the measured Kendall-τ values are 0.74 and 0.62, exceeding the corresponding predicted lower bounds of 0.60 and 0.46. Joint stratification reduces proxy-score variance by 41% relative to random evaluation. Selective teacher mismatch, in contrast, increases differential bias and reduces Kendall-τ to 0.41. Under a constrained evaluation budget, TGL-NSGA-II achieves the largest mean hypervolume and smallest generational distance on keyword spotting, records the lowest mean false-positive rate on BirdCLEF, and runs 2.2x faster than full NSGA-II. These guarantees apply to population-level low-fidelity evaluation and do not establish convergence of the complete evolutionary trajectory.