# Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach

arXiv: https://arxiv.org/abs/2403.11380 · HTML: https://mlsanity.com/paper/2403.11380/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [Neural Architecture Search without Training](https://mlsanity.com/paper/2006.04647/) (2020)
- [Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://mlsanity.com/paper/1904.12760/) (2019)
- [Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours](https://mlsanity.com/paper/1904.02877/) (2019)
- [GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet](https://mlsanity.com/paper/2003.11236/) (2020)
- [AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling](https://mlsanity.com/paper/2011.09011/) (2020)
- [Densely Connected Search Space for More Flexible Neural Architecture Search](https://mlsanity.com/paper/1906.09607/) (2019)
- [Few-shot Neural Architecture Search](https://mlsanity.com/paper/2006.06863/) (2020)
- [Generalizing Few-Shot NAS with Gradient Matching](https://mlsanity.com/paper/2203.15207/) (2022)
- [K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets](https://mlsanity.com/paper/2106.06442/) (2021)
- [Analyzing and Mitigating Interference in Neural Architecture Search](https://mlsanity.com/paper/2108.12821/) (2021)
- [EZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoring](https://mlsanity.com/paper/2209.07413/) (2022)
- [DropNAS: Grouped Operation Dropout for Differentiable Architecture Search](https://mlsanity.com/paper/2201.11679/) (2022)

## Built on by (most cited first, 4 of 4)

- [Meta knowledge assisted Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2504.21545/) (2025)
- [HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning](https://mlsanity.com/paper/2412.10723/) (2024)
- [Neural Architecture Search by Learning a Hierarchical Search Space](https://mlsanity.com/paper/2503.21061/) (2025)
- [LESS: Lightweight Evolutionary Supernet Search in Minutes](https://mlsanity.com/paper/2610.01468/) (2026)

## Abstract (arXiv)

Supernet is a core component in many recent Neural Architecture Search (NAS) methods. It not only helps embody the search space but also provides a (relative) estimation of the final performance of candidate architectures. Thus, it is critical that the top architectures ranked by a supernet should be consistent with those ranked by true performance, which is known as the order-preserving ability. In this work, we analyze the order-preserving ability on the whole search space (global) and a sub-space of top architectures (local), and empirically show that the local order-preserving for current two-stage NAS methods still need to be improved. To rectify this, we propose a novel concept of Supernet Shifting, a refined search strategy combining architecture searching with supernet fine-tuning. Specifically, apart from evaluating, the training loss is also accumulated in searching and the supernet is updated every iteration. Since superior architectures are sampled more frequently in evolutionary searching, the supernet is encouraged to focus on top architectures, thus improving local order-preserving. Besides, a pre-trained supernet is often un-reusable for one-shot methods. We show that Supernet Shifting can fulfill transferring supernet to a new dataset. Specifically, the last classifier layer will be unset and trained through evolutionary searching. Comprehensive experiments show that our method has better order-preserving ability and can find a dominating architecture. Moreover, the pre-trained supernet can be easily transferred into a new dataset with no loss of performance.