mlsanityData to 14 Oct 2026

How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS

arXiv 2003.04276 ↗First posted 9 Mar 2020cs.LG cs.CV stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since heuristics and hyperparameters substantially vary across different methods, a fair comparison between them can only be achieved by systematically analyzing the influence of these factors. In this paper, we therefore provide a systematic evaluation of the heuristics and hyperparameters that are frequently employed by weight-sharing NAS algorithms. Our analysis uncovers that some commonly-used heuristics for super-net training negatively impact the correlation between super-net and stand-alone performance, and evidences the strong influence of certain hyperparameters and architectural choices. Our code and experiments set a strong and reproducible baseline that future works can build on.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 3; 2023: 2; 2024: 2; 2025: 0; 2026: 1.

17 papers.

Sort:

By area:

  1. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
  2. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  3. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 90
  4. 2022NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easycited by 36
  5. 2021HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Searchcited by 20
  6. 2022Neural Architecture Search for Dense Prediction Tasks in Computer Visioncited by 12
  7. 2021Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Searchcited by 8
  8. 2021Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metricscited by 7
  9. 2022GPUNet: Searching the Deployable Convolution Neural Networks for GPUscited by 7
  10. 2023Transfer-Once-For-All: AI Model Optimization for Edgecited by 4
  11. 2021EfficientTDNN: Efficient Architecture Search for Speaker Recognitioncited by 3
  12. 2024Designing a Classifier for Active Fire Detection from Multispectral Satellite Imagery Using Neural Architecture Searchcited by 2
5 more
  1. 2026Bilevel Optimization for Neural Architecture Searchcited by 2
  2. 2021AutoHEnsGNN: Winning Solution to AutoGraph Challenge for KDD Cup 2020cited by 1
  3. 2024Multi-Objective Neural Architecture Search by Learning Search Space Partitionscited by 1
  4. 2021Conditional super-network weightscited by 0
  5. 2021Learning to Rank Ace Neural Architectures via Normalized Discounted Cumulative Gaincited by 0