# Neural Architecture Generator Optimization

arXiv: https://arxiv.org/abs/2004.01395 · HTML: https://mlsanity.com/paper/2004.01395/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2020.

## Built on by (most cited first, 30 of 30)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2301.12132/) (2023)
- [NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy](https://mlsanity.com/paper/2201.13396/) (2022)
- [CATE: Computation-aware Neural Architecture Encoding with Transformers](https://mlsanity.com/paper/2102.07108/) (2021)
- [On Redundancy and Diversity in Cell-based Neural Architecture Search](https://mlsanity.com/paper/2203.08887/) (2022)
- [Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning](https://mlsanity.com/paper/2310.12774/) (2023)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?](https://mlsanity.com/paper/2205.11656/) (2022)
- [A Survey on Computationally Efficient Neural Architecture Search](https://mlsanity.com/paper/2206.01520/) (2022)
- [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://mlsanity.com/paper/2211.01842/) (2022)
- [Adversarial Branch Architecture Search for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2102.06679/) (2021)
- [DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture](https://mlsanity.com/paper/2109.05765/) (2021)
- [Approximate Neural Architecture Search via Operation Distribution Learning](https://mlsanity.com/paper/2111.04670/) (2021)
- [FocusFormer: Focusing on What We Need via Architecture Sampler](https://mlsanity.com/paper/2208.10861/) (2022)
- [GraphPNAS: Learning Distribution of Good Neural Architectures via Deep Graph Generative Models](https://mlsanity.com/paper/2211.15155/) (2022)
- [einspace: Searching for Neural Architectures from Fundamental Operations](https://mlsanity.com/paper/2405.20838/) (2024)
- [Bag of Tricks for Neural Architecture Search](https://mlsanity.com/paper/2107.03719/) (2021)
- [Pareto-aware Neural Architecture Generation for Diverse Computational Budgets](https://mlsanity.com/paper/2210.07634/) (2022)
- [Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets](https://mlsanity.com/paper/2103.00219/) (2021)
- [AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family](https://mlsanity.com/paper/2111.03555/) (2021)
- [Guided Evolutionary Neural Architecture Search With Efficient Performance Estimation](https://mlsanity.com/paper/2208.06475/) (2022)
- [Neural Architecture Search: Two Constant Shared Weights Initialisations](https://mlsanity.com/paper/2302.04406/) (2023)
- [ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures](https://mlsanity.com/paper/2510.04938/) (2025)
- [Agentic Neural Architecture Search](https://mlsanity.com/paper/2607.07984/) (2026)
- [Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints](https://mlsanity.com/paper/2108.01224/) (2021)
- [Network Graph Based Neural Architecture Search](https://mlsanity.com/paper/2112.07805/) (2021)
- [UnrealNAS: Can We Search Neural Architectures with Unreal Data?](https://mlsanity.com/paper/2205.02162/) (2022)
- [CSCO: Connectivity Search of Convolutional Operators](https://mlsanity.com/paper/2404.17152/) (2024)
- [Structural damage detection via hierarchical damage information with volumetric assessment](https://mlsanity.com/paper/2407.19694/) (2024)
- [Self-Auditing Parameter-Efficient Fine-Tuning for Few-Shot 3D Medical Image Segmentation](https://mlsanity.com/paper/2603.05822/) (2026)

## Abstract (arXiv)

Neural Architecture Search (NAS) was first proposed to achieve state-of-the-art performance through the discovery of new architecture patterns, without human intervention. An over-reliance on expert knowledge in the search space design has however led to increased performance (local optima) without significant architectural breakthroughs, thus preventing truly novel solutions from being reached. In this work we 1) are the first to investigate casting NAS as a problem of finding the optimal network generator and 2) we propose a new, hierarchical and graph-based search space capable of representing an extremely large variety of network types, yet only requiring few continuous hyper-parameters. This greatly reduces the dimensionality of the problem, enabling the effective use of Bayesian Optimisation as a search strategy. At the same time, we expand the range of valid architectures, motivating a multi-objective learning approach. We demonstrate the effectiveness of this strategy on six benchmark datasets and show that our search space generates extremely lightweight yet highly competitive models.