# XNAS: Neural Architecture Search with Expert Advice

arXiv: https://arxiv.org/abs/1906.08031 · HTML: https://mlsanity.com/paper/1906.08031/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2019.

## Built on by (most cited first, 30 of 35)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.15954/) (2021)
- [Searching the Search Space of Vision Transformer](https://mlsanity.com/paper/2111.14725/) (2021)
- [HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search](https://mlsanity.com/paper/2102.11646/) (2021)
- [BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer](https://mlsanity.com/paper/2105.08952/) (2021)
- [One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking](https://mlsanity.com/paper/2104.00597/) (2021)
- [DropNAS: Grouped Operation Dropout for Differentiable Architecture Search](https://mlsanity.com/paper/2201.11679/) (2022)
- [iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization](https://mlsanity.com/paper/2108.11014/) (2021)
- [Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search](https://mlsanity.com/paper/2104.05309/) (2021)
- [Search For Deep Graph Neural Networks](https://mlsanity.com/paper/2109.10047/) (2021)
- [BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule](https://mlsanity.com/paper/2111.13204/) (2021)
- [Manas: Mining Software Repositories to Assist AutoML](https://mlsanity.com/paper/2112.03395/) (2021)
- [A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/2101.04243/) (2021)
- [Multiplicative Reweighting for Robust Neural Network Optimization](https://mlsanity.com/paper/2102.12192/) (2021)
- [Learn Basic Skills and Reuse: Modularized Adaptive Neural Architecture Search (MANAS)](https://mlsanity.com/paper/2208.11083/) (2022)
- [Video Action Recognition Collaborative Learning with Dynamics via PSO-ConvNet Transformer](https://mlsanity.com/paper/2302.09187/) (2023)
- [Weight-Entanglement Meets Gradient-Based Neural Architecture Search](https://mlsanity.com/paper/2312.10440/) (2023)
- [FreeAugment: Data Augmentation Search Across All Degrees of Freedom](https://mlsanity.com/paper/2409.04820/) (2024)
- [Integrating Large Circular Kernels into CNNs through Neural Architecture Search](https://mlsanity.com/paper/2107.02451/) (2021)
- [An Analysis of Super-Net Heuristics in Weight-Sharing NAS](https://mlsanity.com/paper/2110.01154/) (2021)
- [BINAS: Bilinear Interpretable Neural Architecture Search](https://mlsanity.com/paper/2110.12399/) (2021)
- [U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search](https://mlsanity.com/paper/2203.12412/) (2022)
- [ApproxDARTS: Differentiable Neural Architecture Search with Approximate Multipliers](https://mlsanity.com/paper/2404.08002/) (2024)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [MS-DARTS: Mean-Shift Based Differentiable Architecture Search](https://mlsanity.com/paper/2108.09996/) (2021)
- [Neural Architecture Search for Inversion](https://mlsanity.com/paper/2201.01772/) (2022)
- [Approximating Permutations with Neural Network Components for Travelling Photographer Problem](https://mlsanity.com/paper/2205.00242/) (2022)

## Abstract (arXiv)

This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. Unlike previous search relaxations, that require hard pruning of architectures, our method is designed to dynamically wipe out inferior architectures and enhance superior ones. It achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates, based on the amount of information carried by the backward gradients. Experiments show that our algorithm achieves a strong performance over several image classification datasets. Specifically, it obtains an error rate of 1.6% for CIFAR-10, 24% for ImageNet under mobile settings, and achieves state-of-the-art results on three additional datasets.