# Evolving Search Space for Neural Architecture Search

arXiv: https://arxiv.org/abs/2011.10904 · HTML: https://mlsanity.com/paper/2011.10904/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2020.

## Built on by (most cited first, 23 of 23)

- [Can GPT-4 Perform Neural Architecture Search?](https://mlsanity.com/paper/2304.10970/) (2023)
- [GLiT: Neural Architecture Search for Global and Local Image Transformer](https://mlsanity.com/paper/2107.02960/) (2021)
- [Searching the Search Space of Vision Transformer](https://mlsanity.com/paper/2111.14725/) (2021)
- [PSViT: Better Vision Transformer via Token Pooling and Attention Sharing](https://mlsanity.com/paper/2108.03428/) (2021)
- [BN-NAS: Neural Architecture Search with Batch Normalization](https://mlsanity.com/paper/2108.07375/) (2021)
- [FBNetV5: Neural Architecture Search for Multiple Tasks in One Run](https://mlsanity.com/paper/2111.10007/) (2021)
- [GreedyNASv2: Greedier Search with a Greedy Path Filter](https://mlsanity.com/paper/2111.12609/) (2021)
- [Automated Design of Metaheuristic Algorithms: A Survey](https://mlsanity.com/paper/2303.06532/) (2023)
- [ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devices](https://mlsanity.com/paper/2303.09730/) (2023)
- [DiffPrep: Differentiable Data Preprocessing Pipeline Search for Learning over Tabular Data](https://mlsanity.com/paper/2308.10915/) (2023)
- [Automated Learning for Deformable Medical Image Registration by Jointly Optimizing Network Architectures and Objective Functions](https://mlsanity.com/paper/2203.06810/) (2022)
- [einspace: Searching for Neural Architectures from Fundamental Operations](https://mlsanity.com/paper/2405.20838/) (2024)
- [Exploring the Intersection between Neural Architecture Search and Continual Learning](https://mlsanity.com/paper/2206.05625/) (2022)
- [Neural Fine-Tuning Search for Few-Shot Learning](https://mlsanity.com/paper/2306.09295/) (2023)
- [Reducing Neural Architecture Search Spaces with Training-Free Statistics and Computational Graph Clustering](https://mlsanity.com/paper/2204.14103/) (2022)
- [Integrating Large Circular Kernels into CNNs through Neural Architecture Search](https://mlsanity.com/paper/2107.02451/) (2021)
- [LISSNAS: Locality-based Iterative Search Space Shrinkage for Neural Architecture Search](https://mlsanity.com/paper/2307.03110/) (2023)
- [SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference](https://mlsanity.com/paper/2303.08308/) (2023)
- [Multi-Objective Neural Architecture Search by Learning Search Space Partitions](https://mlsanity.com/paper/2406.00291/) (2024)
- [Delta-NAS: Difference of Architecture Encoding for Predictor-based Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2411.14498/) (2024)
- [Heed the Noise in Performance Evaluations in Neural Architecture Search](https://mlsanity.com/paper/2202.02078/) (2022)
- [Tiered Pruning for Efficient Differentialble Inference-Aware Neural Architecture Search](https://mlsanity.com/paper/2209.11785/) (2022)
- [Flexible Channel Dimensions for Differentiable Architecture Search](https://mlsanity.com/paper/2306.08021/) (2023)

## Abstract (arXiv)

The automation of neural architecture design has been a coveted alternative to human experts. Recent works have small search space, which is easier to optimize but has a limited upper bound of the optimal solution. Extra human design is needed for those methods to propose a more suitable space with respect to the specific task and algorithm capacity. To further enhance the degree of automation for neural architecture search, we present a Neural Search-space Evolution (NSE) scheme that iteratively amplifies the results from the previous effort by maintaining an optimized search space subset. This design minimizes the necessity of a well-designed search space. We further extend the flexibility of obtainable architectures by introducing a learnable multi-branch setting. By employing the proposed method, a consistent performance gain is achieved during a progressive search over upcoming search spaces. We achieve 77.3% top-1 retrain accuracy on ImageNet with 333M FLOPs, which yielded a state-of-the-art performance among previous auto-generated architectures that do not involve knowledge distillation or weight pruning. When the latency constraint is adopted, our result also performs better than the previous best-performing mobile models with a 77.9% Top-1 retrain accuracy.