# Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search

arXiv: https://arxiv.org/abs/1905.08537 · HTML: https://mlsanity.com/paper/1905.08537/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2019.

## Built on by (most cited first, 23 of 23)

- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [Search to aggregate neighborhood for graph neural network](https://mlsanity.com/paper/2104.06608/) (2021)
- [Generalizing Few-Shot NAS with Gradient Matching](https://mlsanity.com/paper/2203.15207/) (2022)
- [Controllable Dynamic Multi-Task Architectures](https://mlsanity.com/paper/2203.14949/) (2022)
- [You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient](https://mlsanity.com/paper/2106.02435/) (2021)
- [Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift](https://mlsanity.com/paper/2108.09671/) (2021)
- [Searching to Sparsify Tensor Decomposition for N-ary Relational Data](https://mlsanity.com/paper/2104.10625/) (2021)
- [NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters](https://mlsanity.com/paper/2110.10165/) (2021)
- [DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers](https://mlsanity.com/paper/2109.10060/) (2021)
- [DATA: Domain-Aware and Task-Aware Self-supervised Learning](https://mlsanity.com/paper/2203.09041/) (2022)
- [Customized Subgraph Selection and Encoding for Drug-drug Interaction Prediction](https://mlsanity.com/paper/2411.01535/) (2024)
- [Single-DARTS: Towards Stable Architecture Search](https://mlsanity.com/paper/2108.08128/) (2021)
- [Neural Architecture Search for Improving Latency-Accuracy Trade-off in Split Computing](https://mlsanity.com/paper/2208.13968/) (2022)
- [Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architectures](https://mlsanity.com/paper/2106.07211/) (2021)
- [Efficient Search of Multiple Neural Architectures with Different Complexities via Importance Sampling](https://mlsanity.com/paper/2207.10334/) (2022)
- [Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methods](https://mlsanity.com/paper/2403.08352/) (2024)
- [Aux-NAS: Exploiting Auxiliary Labels with Negligibly Extra Inference Cost](https://mlsanity.com/paper/2405.05695/) (2024)
- [Multi-Objective Neural Architecture Search by Learning Search Space Partitions](https://mlsanity.com/paper/2406.00291/) (2024)
- [ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces](https://mlsanity.com/paper/2101.07415/) (2021)
- [Is Differentiable Architecture Search truly a One-Shot Method?](https://mlsanity.com/paper/2108.05647/) (2021)
- [DMTG: One-Shot Differentiable Multi-Task Grouping](https://mlsanity.com/paper/2407.05082/) (2024)
- [Neural Architecture Search of Sample Reweighting Networks for Complex Distribution Shift](https://mlsanity.com/paper/2606.22991/) (2026)

## Abstract (arXiv)

High sensitivity of neural architecture search (NAS) methods against their input such as step-size (i.e., learning rate) and search space prevents practitioners from applying them out-of-the-box to their own problems, albeit its purpose is to automate a part of tuning process. Aiming at a fast, robust, and widely-applicable NAS, we develop a generic optimization framework for NAS. We turn a coupled optimization of connection weights and neural architecture into a differentiable optimization by means of stochastic relaxation. It accepts arbitrary search space (widely-applicable) and enables to employ a gradient-based simultaneous optimization of weights and architecture (fast). We propose a stochastic natural gradient method with an adaptive step-size mechanism built upon our theoretical investigation (robust). Despite its simplicity and no problem-dependent parameter tuning, our method exhibited near state-of-the-art performances with low computational budgets both on image classification and inpainting tasks.