# Neural Architecture Search as Sparse Supernet

arXiv: https://arxiv.org/abs/2007.16112 · HTML: https://mlsanity.com/paper/2007.16112/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results](https://mlsanity.com/paper/2205.05675/) (2022)
- [Neural Architecture Search via Bregman Iterations](https://mlsanity.com/paper/2106.02479/) (2021)
- [QuantNAS for super resolution: searching for efficient quantization-friendly architectures against quantization noise](https://mlsanity.com/paper/2208.14839/) (2022)
- [A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search](https://mlsanity.com/paper/2204.03916/) (2022)
- [Rethink DARTS Search Space and Renovate a New Benchmark](https://mlsanity.com/paper/2306.06852/) (2023)
- [Fine-grained Token Allocation Via Operation Pruning for Efficient MLLMs](https://mlsanity.com/paper/2507.02909/) (2025)
- [Learning k-Level Structured Sparse Neural Networks Using Group Envelope Regularization](https://mlsanity.com/paper/2212.12921/) (2022)
- [Small Temperature is All You Need for Differentiable Architecture Search](https://mlsanity.com/paper/2306.06855/) (2023)
- [Is Differentiable Architecture Search truly a One-Shot Method?](https://mlsanity.com/paper/2108.05647/) (2021)
- [Physics-Informed Machine Learning in Prognostics and Health Management: A Systematic Literature Review](https://mlsanity.com/paper/2608.10047/) (2026)

## Abstract (arXiv)

This paper aims at enlarging the problem of Neural Architecture Search (NAS) from Single-Path and Multi-Path Search to automated Mixed-Path Search. In particular, we model the NAS problem as a sparse supernet using a new continuous architecture representation with a mixture of sparsity constraints. The sparse supernet enables us to automatically achieve sparsely-mixed paths upon a compact set of nodes. To optimize the proposed sparse supernet, we exploit a hierarchical accelerated proximal gradient algorithm within a bi-level optimization framework. Extensive experiments on Convolutional Neural Network and Recurrent Neural Network search demonstrate that the proposed method is capable of searching for compact, general and powerful neural architectures.