# Can weight sharing outperform random architecture search? An investigation with TuNAS

arXiv: https://arxiv.org/abs/2008.06120 · HTML: https://mlsanity.com/paper/2008.06120/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2020.

## Built on by (most cited first, 30 of 68)

- [Primer: Searching for Efficient Transformers for Language Modeling](https://mlsanity.com/paper/2109.08668/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [MobileNetV4 -- Universal Models for the Mobile Ecosystem](https://mlsanity.com/paper/2404.10518/) (2024)
- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Packed-Ensembles for Efficient Uncertainty Estimation](https://mlsanity.com/paper/2210.09184/) (2022)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [Searching for Fast Model Families on Datacenter Accelerators](https://mlsanity.com/paper/2102.05610/) (2021)
- [CATE: Computation-aware Neural Architecture Encoding with Transformers](https://mlsanity.com/paper/2102.07108/) (2021)
- [PyGlove: Symbolic Programming for Automated Machine Learning](https://mlsanity.com/paper/2101.08809/) (2021)
- [Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift](https://mlsanity.com/paper/2108.09671/) (2021)
- [HyperSPNs: Compact and Expressive Probabilistic Circuits](https://mlsanity.com/paper/2112.00914/) (2021)
- [Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems](https://mlsanity.com/paper/2305.12102/) (2023)
- [NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization](https://mlsanity.com/paper/2104.00031/) (2021)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Rethinking Co-design of Neural Architectures and Hardware Accelerators](https://mlsanity.com/paper/2102.08619/) (2021)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models](https://mlsanity.com/paper/2405.10299/) (2024)
- [Generative Adversarial Neural Architecture Search](https://mlsanity.com/paper/2105.09356/) (2021)
- [Searching for Efficient Multi-Stage Vision Transformers](https://mlsanity.com/paper/2109.00642/) (2021)
- [Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search](https://mlsanity.com/paper/2104.05309/) (2021)
- [One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2111.01203/) (2021)
- [Fisher Task Distance and Its Application in Neural Architecture Search](https://mlsanity.com/paper/2103.12827/) (2021)
- [NASRec: Weight Sharing Neural Architecture Search for Recommender Systems](https://mlsanity.com/paper/2207.07187/) (2022)
- [Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices](https://mlsanity.com/paper/2310.07217/) (2023)
- [L2NAS: Learning to Optimize Neural Architectures via Continuous-Action Reinforcement Learning](https://mlsanity.com/paper/2109.12425/) (2021)
- [TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets](https://mlsanity.com/paper/2204.07615/) (2022)

## Abstract (arXiv)

Efficient Neural Architecture Search methods based on weight sharing have shown good promise in democratizing Neural Architecture Search for computer vision models. There is, however, an ongoing debate whether these efficient methods are significantly better than random search. Here we perform a thorough comparison between efficient and random search methods on a family of progressively larger and more challenging search spaces for image classification and detection on ImageNet and COCO. While the efficacies of both methods are problem-dependent, our experiments demonstrate that there are large, realistic tasks where efficient search methods can provide substantial gains over random search. In addition, we propose and evaluate techniques which improve the quality of searched architectures and reduce the need for manual hyper-parameter tuning. Source code and experiment data are available at https://github.com/google-research/google-research/tree/master/tunas