# Neural Architecture Transfer 2: A Paradigm for Improving Efficiency in Multi-Objective Neural Architecture Search

arXiv: https://arxiv.org/abs/2307.00960 · HTML: https://mlsanity.com/paper/2307.00960/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [Compute and Energy Consumption Trends in Deep Learning Inference](https://mlsanity.com/paper/2109.05472/) (2021)
- [Neural Architecture Transfer](https://mlsanity.com/paper/2005.05859/) (2020)
- [Overcoming Multi-Model Forgetting](https://mlsanity.com/paper/1902.08232/) (2019)
- [Pareto-wise Ranking Classifier for Multi-objective Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2109.07582/) (2021)
- [POPNASv3: a Pareto-Optimal Neural Architecture Search Solution for Image and Time Series Classification](https://mlsanity.com/paper/2212.06735/) (2022)
- [POPNASv2: An Efficient Multi-Objective Neural Architecture Search Technique](https://mlsanity.com/paper/2210.02959/) (2022)
- [Anticipate, Ensemble and Prune: Improving Convolutional Neural Networks via Aggregated Early Exits](https://mlsanity.com/paper/2301.12168/) (2023)
- [Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits](https://mlsanity.com/paper/2302.01888/) (2023)

## Built on by (most cited first, 1 of 1)

- [POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator](https://mlsanity.com/paper/2409.20447/) (2024)

## Abstract (arXiv)

Deep learning is increasingly impacting various aspects of contemporary society. Artificial neural networks have emerged as the dominant models for solving an expanding range of tasks. The introduction of Neural Architecture Search (NAS) techniques, which enable the automatic design of task-optimal networks, has led to remarkable advances. However, the NAS process is typically associated with long execution times and significant computational resource requirements. Once-For-All (OFA) and its successor, Once-For-All-2 (OFAv2), have been developed to mitigate these challenges. While maintaining exceptional performance and eliminating the need for retraining, they aim to build a single super-network model capable of directly extracting sub-networks satisfying different constraints. Neural Architecture Transfer (NAT) was developed to maximise the effectiveness of extracting sub-networks from a super-network. In this paper, we present NATv2, an extension of NAT that improves multi-objective search algorithms applied to dynamic super-network architectures. NATv2 achieves qualitative improvements in the extractable sub-networks by exploiting the improved super-networks generated by OFAv2 and incorporating new policies for initialisation, pre-processing and updating its networks archive. In addition, a post-processing pipeline based on fine-tuning is introduced. Experimental results show that NATv2 successfully improves NAT and is highly recommended for investigating high-performance architectures with a minimal number of parameters.