# NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization

arXiv: https://arxiv.org/abs/2104.00031 · HTML: https://mlsanity.com/paper/2104.00031/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 30 of 35)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Random Search and Reproducibility for Neural Architecture Search](https://mlsanity.com/paper/1902.07638/) (2019)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions](https://mlsanity.com/paper/2004.05565/) (2020)
- [NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications](https://mlsanity.com/paper/1804.03230/) (2018)
- [FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search](https://mlsanity.com/paper/1907.01845/) (2019)
- [MixConv: Mixed Depthwise Convolutional Kernels](https://mlsanity.com/paper/1907.09595/) (2019)
- [MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks](https://mlsanity.com/paper/1711.06798/) (2017)
- [FastDepth: Fast Monocular Depth Estimation on Embedded Systems](https://mlsanity.com/paper/1903.03273/) (2019)
- [Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours](https://mlsanity.com/paper/1904.02877/) (2019)
- [ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation](https://mlsanity.com/paper/1812.08934/) (2018)

## Built on by (most cited first, 13 of 13)

- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [SPDY: Accurate Pruning with Speedup Guarantees](https://mlsanity.com/paper/2201.13096/) (2022)
- [Partial Variable Training for Efficient On-Device Federated Learning](https://mlsanity.com/paper/2110.05607/) (2021)
- [Soft Masking for Cost-Constrained Channel Pruning](https://mlsanity.com/paper/2211.02206/) (2022)
- [Searching for Efficient Multi-Stage Vision Transformers](https://mlsanity.com/paper/2109.00642/) (2021)
- [Online Model Compression for Federated Learning with Large Models](https://mlsanity.com/paper/2205.03494/) (2022)
- [Pareto-aware Neural Architecture Generation for Diverse Computational Budgets](https://mlsanity.com/paper/2210.07634/) (2022)
- [CPrune: Compiler-Informed Model Pruning for Efficient Target-Aware DNN Execution](https://mlsanity.com/paper/2207.01260/) (2022)
- [Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey](https://mlsanity.com/paper/2208.10498/) (2022)
- [From the Pursuit of Universal AGI Architecture to Systematic Approach to Heterogenous AGI: Addressing Alignment, Energy, & AGI Grand Challenges](https://mlsanity.com/paper/2310.15274/) (2023)
- [Exploring Deep-to-Shallow Transformable Neural Networks for Intelligent Embedded Systems](https://mlsanity.com/paper/2512.19731/) (2025)

## Abstract (arXiv)

Neural architecture search (NAS) typically consists of three main steps: training a super-network, training and evaluating sampled deep neural networks (DNNs), and training the discovered DNN. Most of the existing efforts speed up some steps at the cost of a significant slowdown of other steps or sacrificing the support of non-differentiable search metrics. The unbalanced reduction in the time spent per step limits the total search time reduction, and the inability to support non-differentiable search metrics limits the performance of discovered DNNs. In this paper, we present NetAdaptV2 with three innovations to better balance the time spent for each step while supporting non-differentiable search metrics. First, we propose channel-level bypass connections that merge network depth and layer width into a single search dimension to reduce the time for training and evaluating sampled DNNs. Second, ordered dropout is proposed to train multiple DNNs in a single forward-backward pass to decrease the time for training a super-network. Third, we propose the multi-layer coordinate descent optimizer that considers the interplay of multiple layers in each iteration of optimization to improve the performance of discovered DNNs while supporting non-differentiable search metrics. With these innovations, NetAdaptV2 reduces the total search time by up to 5.8× on ImageNet and 2.4× on NYU Depth V2, respectively, and discovers DNNs with better accuracy-latency/accuracy-MAC trade-offs than state-of-the-art NAS works. Moreover, the discovered DNN outperforms NAS-discovered MobileNetV3 by 1.8% higher top-1 accuracy with the same latency. The project website is http://netadapt.mit.edu.