# Deeper Insights into Weight Sharing in Neural Architecture Search

arXiv: https://arxiv.org/abs/2001.01431 · HTML: https://mlsanity.com/paper/2001.01431/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2020.

## Built on by (most cited first, 24 of 24)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [How Powerful are Performance Predictors in Neural Architecture Search?](https://mlsanity.com/paper/2104.01177/) (2021)
- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [Generalizing Few-Shot NAS with Gradient Matching](https://mlsanity.com/paper/2203.15207/) (2022)
- [Analyzing and Mitigating Interference in Neural Architecture Search](https://mlsanity.com/paper/2108.12821/) (2021)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks](https://mlsanity.com/paper/2301.09422/) (2023)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving](https://mlsanity.com/paper/2108.08019/) (2021)
- [Speeding up Deep Model Training by Sharing Weights and Then Unsharing](https://mlsanity.com/paper/2110.03848/) (2021)
- [Automatic Relation-aware Graph Network Proliferation](https://mlsanity.com/paper/2205.15678/) (2022)
- [Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning](https://mlsanity.com/paper/2407.01320/) (2024)
- [AutoSpace: Neural Architecture Search with Less Human Interference](https://mlsanity.com/paper/2103.11833/) (2021)
- [QuickLoc: Adaptive Deep-Learning for Fast Indoor Localization with Mobile Devices](https://mlsanity.com/paper/2104.07521/) (2021)
- [Rethink DARTS Search Space and Renovate a New Benchmark](https://mlsanity.com/paper/2306.06852/) (2023)
- [Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization](https://mlsanity.com/paper/2306.06858/) (2023)
- [AutoHEnsGNN: Winning Solution to AutoGraph Challenge for KDD Cup 2020](https://mlsanity.com/paper/2111.12952/) (2021)
- [Improve Ranking Correlation of Super-net through Training Scheme from One-shot NAS to Few-shot NAS](https://mlsanity.com/paper/2206.05896/) (2022)
- [Conditional super-network weights](https://mlsanity.com/paper/2104.11522/) (2021)
- [Learning to Rank Ace Neural Architectures via Normalized Discounted Cumulative Gain](https://mlsanity.com/paper/2108.03001/) (2021)
- [Delve into the Performance Degradation of Differentiable Architecture Search](https://mlsanity.com/paper/2109.13466/) (2021)
- [Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks](https://mlsanity.com/paper/2307.08013/) (2023)
- [CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer](https://mlsanity.com/paper/2511.14111/) (2025)
- [Layered Quantum Architecture Search for 3D Point Cloud Classification](https://mlsanity.com/paper/2603.20024/) (2026)

## Abstract (arXiv)

With the success of deep neural networks, Neural Architecture Search (NAS) as a way of automatic model design has attracted wide attention. As training every child model from scratch is very time-consuming, recent works leverage weight-sharing to speed up the model evaluation procedure. These approaches greatly reduce computation by maintaining a single copy of weights on the super-net and share the weights among every child model. However, weight-sharing has no theoretical guarantee and its impact has not been well studied before. In this paper, we conduct comprehensive experiments to reveal the impact of weight-sharing: (1) The best-performing models from different runs or even from consecutive epochs within the same run have significant variance; (2) Even with high variance, we can extract valuable information from training the super-net with shared weights; (3) The interference between child models is a main factor that induces high variance; (4) Properly reducing the degree of weight sharing could effectively reduce variance and improve performance.