# A Data-driven Approach to Neural Architecture Search Initialization

arXiv: https://arxiv.org/abs/2111.03524 · HTML: https://mlsanity.com/paper/2111.03524/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 60 references that are in the map.
- First posted on arXiv 5 Nov 2021.

## Builds on (most cited first, 19 of 19)

- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [Evolving Deep Neural Networks](https://mlsanity.com/paper/1703.00548/) (2017)
- [Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition](https://mlsanity.com/paper/2102.01063/) (2021)
- [NAS evaluation is frustratingly hard](https://mlsanity.com/paper/1912.12522/) (2019)
- [Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search](https://mlsanity.com/paper/1901.07261/) (2019)
- [NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search](https://mlsanity.com/paper/2001.10422/) (2020)
- [NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing](https://mlsanity.com/paper/2006.07116/) (2020)
- [A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS](https://mlsanity.com/paper/2004.01899/) (2020)
- [Evolutionary Generative Adversarial Networks](https://mlsanity.com/paper/1803.00657/) (2018)
- [Metaheuristic Design of Feedforward Neural Networks: A Review of Two Decades of Research](https://mlsanity.com/paper/1705.05584/) (2017)
- [Investigating Recurrent Neural Network Memory Structures using Neuro-Evolution](https://mlsanity.com/paper/1902.02390/) (2019)
- [Bayesian Neural Architecture Search using A Training-Free Performance Metric](https://mlsanity.com/paper/2001.10726/) (2020)
- [Low-Cost Recurrent Neural Network Expected Performance Evaluation](https://mlsanity.com/paper/1805.07159/) (2018)
- [Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems](https://mlsanity.com/paper/2111.01584/) (2021)

## Built on by (most cited first, 2 of 2)

- [Landscape of Neural Architecture Search across sensors: how much do they differ ?](https://mlsanity.com/paper/2201.06321/) (2022)
- [SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Search](https://mlsanity.com/paper/2402.13204/) (2024)

## Abstract (arXiv)

Algorithmic design in neural architecture search (NAS) has received a lot of attention, aiming to improve performance and reduce computational cost. Despite the great advances made, few authors have proposed to tailor initialization techniques for NAS. However, literature shows that a good initial set of solutions facilitate finding the optima. Therefore, in this study, we propose a data-driven technique to initialize a population-based NAS algorithm. Particularly, we proposed a two-step methodology. First, we perform a calibrated clustering analysis of the search space, and second, we extract the centroids and use them to initialize a NAS algorithm. We benchmark our proposed approach against random and Latin hypercube sampling initialization using three population-based algorithms, namely a genetic algorithm, evolutionary algorithm, and aging evolution, on CIFAR-10. More specifically, we use NAS-Bench-101 to leverage the availability of NAS benchmarks. The results show that compared to random and Latin hypercube sampling, the proposed initialization technique enables achieving significant long-term improvements for two of the search baselines, and sometimes in various search scenarios (various training budgets). Moreover, we analyze the distributions of solutions obtained and find that that the population provided by the data-driven initialization technique enables retrieving local optima (maxima) of high fitness and similar configurations.