# Few-shot Neural Architecture Search

arXiv: https://arxiv.org/abs/2006.06863 · HTML: https://mlsanity.com/paper/2006.06863/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 30 of 50)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [LLM Performance Predictors are good initializers for Architecture Search](https://mlsanity.com/paper/2310.16712/) (2023)
- [AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search](https://mlsanity.com/paper/2403.19232/) (2024)
- [Generalizing Few-Shot NAS with Gradient Matching](https://mlsanity.com/paper/2203.15207/) (2022)
- [On Redundancy and Diversity in Cell-based Neural Architecture Search](https://mlsanity.com/paper/2203.08887/) (2022)
- [EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search](https://mlsanity.com/paper/2210.06015/) (2022)
- [Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search](https://mlsanity.com/paper/2104.05309/) (2021)
- [RADARS: Memory Efficient Reinforcement Learning Aided Differentiable Neural Architecture Search](https://mlsanity.com/paper/2109.05691/) (2021)
- [One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2111.01203/) (2021)
- [Fisher Task Distance and Its Application in Neural Architecture Search](https://mlsanity.com/paper/2103.12827/) (2021)
- [Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics](https://mlsanity.com/paper/2108.11939/) (2021)
- [GPUNet: Searching the Deployable Convolution Neural Networks for GPUs](https://mlsanity.com/paper/2205.00841/) (2022)
- [Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts](https://mlsanity.com/paper/2306.04845/) (2023)
- [FNAS: Uncertainty-Aware Fast Neural Architecture Search](https://mlsanity.com/paper/2105.11694/) (2021)
- [Learning Deep Morphological Networks with Neural Architecture Search](https://mlsanity.com/paper/2106.07714/) (2021)
- [TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets](https://mlsanity.com/paper/2204.07615/) (2022)
- [PA&DA: Jointly Sampling PAth and DAta for Consistent NAS](https://mlsanity.com/paper/2302.14772/) (2023)
- [CLOSE: Curriculum Learning On the Sharing Extent Towards Better One-shot NAS](https://mlsanity.com/paper/2207.07868/) (2022)
- [NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension](https://mlsanity.com/paper/2211.12759/) (2022)
- [CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework](https://mlsanity.com/paper/2406.01414/) (2024)
- [Bag of Tricks for Neural Architecture Search](https://mlsanity.com/paper/2107.03719/) (2021)
- [Pareto-aware Neural Architecture Generation for Diverse Computational Budgets](https://mlsanity.com/paper/2210.07634/) (2022)
- [Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach](https://mlsanity.com/paper/2403.11380/) (2024)
- [Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers](https://mlsanity.com/paper/2407.18175/) (2024)
- [Customized Subgraph Selection and Encoding for Drug-drug Interaction Prediction](https://mlsanity.com/paper/2411.01535/) (2024)
- [Subnet-Aware Dynamic Supernet Training for Neural Architecture Search](https://mlsanity.com/paper/2503.10740/) (2025)
- [AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models](https://mlsanity.com/paper/2201.12507/) (2022)
- [When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Search](https://mlsanity.com/paper/2204.04918/) (2022)
- [Automated Dominative Subspace Mining for Efficient Neural Architecture Search](https://mlsanity.com/paper/2210.17180/) (2022)

## Abstract (arXiv)

Efficient evaluation of a network architecture drawn from a large search space remains a key challenge in Neural Architecture Search (NAS). Vanilla NAS evaluates each architecture by training from scratch, which gives the true performance but is extremely time-consuming. Recently, one-shot NAS substantially reduces the computation cost by training only one supernetwork, a.k.a. supernet, to approximate the performance of every architecture in the search space via weight-sharing. However, the performance estimation can be very inaccurate due to the co-adaption among operations. In this paper, we propose few-shot NAS that uses multiple supernetworks, called sub-supernet, each covering different regions of the search space to alleviate the undesired co-adaption. Compared to one-shot NAS, few-shot NAS improves the accuracy of architecture evaluation with a small increase of evaluation cost. With only up to 7 sub-supernets, few-shot NAS establishes new SoTAs: on ImageNet, it finds models that reach 80.5% top-1 accuracy at 600 MB FLOPS and 77.5% top-1 accuracy at 238 MFLOPS; on CIFAR10, it reaches 98.72% top-1 accuracy without using extra data or transfer learning. In Auto-GAN, few-shot NAS outperforms the previously published results by up to 20%. Extensive experiments show that few-shot NAS significantly improves various one-shot methods, including 4 gradient-based and 6 search-based methods on 3 different tasks in NasBench-201 and NasBench1-shot-1.