# Probabilistic Neural Architecture Search

arXiv: https://arxiv.org/abs/1902.05116 · HTML: https://mlsanity.com/paper/1902.05116/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2019.

## Built on by (most cited first, 20 of 20)

- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [Generalizing Few-Shot NAS with Gradient Matching](https://mlsanity.com/paper/2203.15207/) (2022)
- [FBNetV5: Neural Architecture Search for Multiple Tasks in One Run](https://mlsanity.com/paper/2111.10007/) (2021)
- [DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture](https://mlsanity.com/paper/2109.05765/) (2021)
- [Approximate Neural Architecture Search via Operation Distribution Learning](https://mlsanity.com/paper/2111.04670/) (2021)
- [Learning from Mistakes -- A Framework for Neural Architecture Search](https://mlsanity.com/paper/2111.06353/) (2021)
- [Contrastive Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2102.10557/) (2021)
- [Searching for Two-Stream Models in Multivariate Space for Video Recognition](https://mlsanity.com/paper/2108.12957/) (2021)
- [Differentiable Architecture Search with Random Features](https://mlsanity.com/paper/2208.08835/) (2022)
- [Learning by Teaching, with Application to Neural Architecture Search](https://mlsanity.com/paper/2103.07009/) (2021)
- [Interleaving Learning, with Application to Neural Architecture Search](https://mlsanity.com/paper/2103.07018/) (2021)
- [Rapid Model Architecture Adaption for Meta-Learning](https://mlsanity.com/paper/2109.04925/) (2021)
- [Width Transfer: On the (In)variance of Width Optimization](https://mlsanity.com/paper/2104.13255/) (2021)
- [Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search](https://mlsanity.com/paper/2112.00275/) (2021)
- [Auto-X3D: Ultra-Efficient Video Understanding via Finer-Grained Neural Architecture Search](https://mlsanity.com/paper/2112.04710/) (2021)
- [Neural Architecture Search for Inversion](https://mlsanity.com/paper/2201.01772/) (2022)
- [Model Architecture Adaption for Bayesian Neural Networks](https://mlsanity.com/paper/2202.04392/) (2022)
- [WPNAS: Neural Architecture Search by jointly using Weight Sharing and Predictor](https://mlsanity.com/paper/2203.02086/) (2022)
- [A Survey on Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2307.09099/) (2023)

## Abstract (arXiv)

In neural architecture search (NAS), the space of neural network architectures is automatically explored to maximize predictive accuracy for a given task. Despite the success of recent approaches, most existing methods cannot be directly applied to large scale problems because of their prohibitive computational complexity or high memory usage. In this work, we propose a Probabilistic approach to neural ARchitecture SEarCh (PARSEC) that drastically reduces memory requirements while maintaining state-of-the-art computational complexity, making it possible to directly search over more complex architectures and larger datasets. Our approach only requires as much memory as is needed to train a single architecture from our search space. This is due to a memory-efficient sampling procedure wherein we learn a probability distribution over high-performing neural network architectures. Importantly, this framework enables us to transfer the distribution of architectures learnt on smaller problems to larger ones, further reducing the computational cost. We showcase the advantages of our approach in applications to CIFAR-10 and ImageNet, where our approach outperforms methods with double its computational cost and matches the performance of methods with costs that are three orders of magnitude larger.