How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictor
Not on the map. Why?
Abstract (arXiv)
Neural architecture search has proven to be a powerful approach to designing and refining neural networks, often boosting their performance and efficiency over manually-designed variations, but comes with computational overhead. While there has been a considerable amount of research focused on lowering the cost of NAS for mainstream tasks, such as image classification, a lot of those improvements stem from the fact that those tasks are well-studied in the broader context. Consequently, applicability of NAS to emerging and under-represented domains is still associated with a relatively high cost and/or uncertainty about the achievable gains. To address this issue, we turn our focus towards the recent growth of publicly available NAS benchmarks in an attempt to extract general NAS knowledge, transferable across different tasks and search spaces. We borrow from the rich field of meta-learning for few-shot adaptation and carefully study applicability of those methods to NAS, with a special focus on the relationship between task-level correlation (domain shift) and predictor transferability; which we deem critical for improving NAS on diverse tasks. In our experiments, we use 6 NAS benchmarks in conjunction, spanning in total 16 NAS settings -- our meta-learning approach not only shows superior (or matching) performance in the cross-validation experiments but also successful extrapolation to a new search space and tasks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 48 of its 60 references that are in the map. First posted on arXiv 30 Nov 2023.
Builds on
48 of 60 references.
References · 48
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
- 2018DARTS: Differentiable Architecture Searchcited by 1,798
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
- 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
- 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
- 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
- 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
- 2018Spherical CNNscited by 456
- 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
- 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
36 more
- 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
- 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
- 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
- 2019ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernelscited by 339
- 2018How to train your MAMLcited by 309
- 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 260
- 2020Neural Architecture Search without Trainingcited by 244
- 2021Zero-Cost Proxies for Lightweight NAScited by 199
- 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 179
- 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
- 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 126
- 2019Neural Predictor for Neural Architecture Searchcited by 118
- 2020BRP-NAS: Prediction-based NAS using GCNscited by 111
- 2021Rethinking Architecture Selection in Differentiable NAScited by 92
- 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 90
- 2019Blockwisely Supervised Neural Architecture Search with Knowledge Distillationcited by 81
- 2019Meta-Learning of Neural Architectures for Few-Shot Learningcited by 73
- 2019SGAS: Sequential Greedy Architecture Searchcited by 69
- 2021TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Searchcited by 58
- 2019Densely Connected Search Space for More Flexible Neural Architecture Searchcited by 52
- 2020Semi-Supervised Neural Architecture Searchcited by 46
- 2020HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Unitscited by 45
- 2020Are Labels Necessary for Neural Architecture Search?cited by 37
- 2021Contrastive Neural Architecture Search with Neural Architecture Comparatorscited by 34
- 2021Rapid Neural Architecture Search by Learning to Generate Graphs from Datasetscited by 33
- 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
- 2020A Semi-Supervised Assessor of Neural Architecturescited by 28
- 2021Stronger NAS with Weaker Predictorscited by 28
- 2018Meta Architecture Searchcited by 19
- 2021Rethinking Neural Operations for Diverse Taskscited by 18
- 2022Efficient Architecture Search for Diverse Taskscited by 18
- 2021HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learningcited by 16
- 2021Generic Neural Architecture Search via Regressioncited by 14
- 2020sEMG Gesture Recognition with a Simple Model of Attentioncited by 10
- 2022BLOX: Macro Neural Architecture Search Benchmark and Algorithmscited by 6
- 2023Dynamic Ensemble of Low-fidelity Experts: Mitigating NAS "Cold-Start"cited by 3
Built on by
Citing papers per year: 2025: 2.
2 papers.