# Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

arXiv: https://arxiv.org/abs/2008.01475 · HTML: https://mlsanity.com/paper/2008.01475/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2020.

## Built on by (most cited first, 30 of 50)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [AutoSNN: Towards Energy-Efficient Spiking Neural Networks](https://mlsanity.com/paper/2201.12738/) (2022)
- [NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy](https://mlsanity.com/paper/2201.13396/) (2022)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [Towards a Unified View on Visual Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2210.00788/) (2022)
- [CATE: Computation-aware Neural Architecture Encoding with Transformers](https://mlsanity.com/paper/2102.07108/) (2021)
- [Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2307.01998/) (2023)
- [Analyzing and Mitigating Interference in Neural Architecture Search](https://mlsanity.com/paper/2108.12821/) (2021)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective](https://mlsanity.com/paper/2112.14531/) (2021)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [Learning Diverse-Structured Networks for Adversarial Robustness](https://mlsanity.com/paper/2102.01886/) (2021)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables](https://mlsanity.com/paper/2303.14506/) (2023)
- [SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS](https://mlsanity.com/paper/2403.04161/) (2024)
- [ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks](https://mlsanity.com/paper/2205.08119/) (2022)
- [Equivariance-aware Architectural Optimization of Neural Networks](https://mlsanity.com/paper/2210.05484/) (2022)
- [DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture](https://mlsanity.com/paper/2109.05765/) (2021)
- [L2NAS: Learning to Optimize Neural Architectures via Continuous-Action Reinforcement Learning](https://mlsanity.com/paper/2109.12425/) (2021)
- [Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning](https://mlsanity.com/paper/2407.01320/) (2024)
- [Evolutionary Multi-objective Architecture Search Framework: Application to COVID-19 3D CT Classification](https://mlsanity.com/paper/2101.10667/) (2021)
- [EAGAN: Efficient Two-stage Evolutionary Architecture Search for GANs](https://mlsanity.com/paper/2111.15097/) (2021)
- [Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search](https://mlsanity.com/paper/2304.05405/) (2023)
- [Do Not Train It: A Linear Neural Architecture Search of Graph Neural Networks](https://mlsanity.com/paper/2305.14065/) (2023)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [Block shuffling learning for Deepfake Detection](https://mlsanity.com/paper/2202.02819/) (2022)
- [Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Models](https://mlsanity.com/paper/2504.20020/) (2025)
- [EfficientTDNN: Efficient Architecture Search for Speaker Recognition](https://mlsanity.com/paper/2103.13581/) (2021)
- [What Is Considered Complete for Visual Recognition?](https://mlsanity.com/paper/2105.13978/) (2021)
- [FEAR: A Simple Lightweight Method to Rank Architectures](https://mlsanity.com/paper/2106.04010/) (2021)

## Abstract (arXiv)

Neural architecture search (NAS) has attracted increasing attentions in both academia and industry. In the early age, researchers mostly applied individual search methods which sample and evaluate the candidate architectures separately and thus incur heavy computational overheads. To alleviate the burden, weight-sharing methods were proposed in which exponentially many architectures share weights in the same super-network, and the costly training procedure is performed only once. These methods, though being much faster, often suffer the issue of instability. This paper provides a literature review on NAS, in particular the weight-sharing methods, and points out that the major challenge comes from the optimization gap between the super-network and the sub-architectures. From this perspective, we summarize existing approaches into several categories according to their efforts in bridging the gap, and analyze both advantages and disadvantages of these methodologies. Finally, we share our opinions on the future directions of NAS and AutoML. Due to the expertise of the authors, this paper mainly focuses on the application of NAS to computer vision problems and may bias towards the work in our group.