# Neural Architecture Transfer

arXiv: https://arxiv.org/abs/2005.05859 · HTML: https://mlsanity.com/paper/2005.05859/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 30 of 60)

- [FaPN: Feature-aligned Pyramid Network for Dense Image Prediction](https://mlsanity.com/paper/2108.07058/) (2021)
- [Source-Free Unsupervised Domain Adaptation: A Survey](https://mlsanity.com/paper/2301.00265/) (2022)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022)
- [Teachers Do More Than Teach: Compressing Image-to-Image Models](https://mlsanity.com/paper/2103.03467/) (2021)
- [BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer](https://mlsanity.com/paper/2105.08952/) (2021)
- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [Meta Navigator: Search for a Good Adaptation Policy for Few-shot Learning](https://mlsanity.com/paper/2109.05749/) (2021)
- [Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment](https://mlsanity.com/paper/2208.04321/) (2022)
- [Design Principle Transfer in Neural Architecture Search via Large Language Models](https://mlsanity.com/paper/2408.11330/) (2024)
- [Memory-Efficient Differentiable Transformer Architecture Search](https://mlsanity.com/paper/2105.14669/) (2021)
- [A Survey on Computationally Efficient Neural Architecture Search](https://mlsanity.com/paper/2206.01520/) (2022)
- [Exploring Robust Architectures for Deep Artificial Neural Networks](https://mlsanity.com/paper/2106.15850/) (2021)
- [Reconsidering CO2 emissions from Computer Vision](https://mlsanity.com/paper/2104.08702/) (2021)
- [AdaXpert: Adapting Neural Architecture for Growing Data](https://mlsanity.com/paper/2107.00254/) (2021)
- [A Survey on Evolutionary Computation for Computer Vision and Image Analysis: Past, Present, and Future Trends](https://mlsanity.com/paper/2209.06399/) (2022)
- [Efficient Adaptive Ensembling for Image Classification](https://mlsanity.com/paper/2206.07394/) (2022)
- [Pareto-wise Ranking Classifier for Multi-objective Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2109.07582/) (2021)
- [A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities](https://mlsanity.com/paper/2205.10358/) (2022)
- [QCQA: Quality and Capacity-aware grouped Query Attention](https://mlsanity.com/paper/2406.10247/) (2024)
- [Generalized Global Ranking-Aware Neural Architecture Ranker for Efficient Image Classifier Search](https://mlsanity.com/paper/2201.12725/) (2022)
- [Surrogate-assisted Multi-objective Neural Architecture Search for Real-time Semantic Segmentation](https://mlsanity.com/paper/2208.06820/) (2022)
- [A Scalable Test Problem Generator for Sequential Transfer Optimization](https://mlsanity.com/paper/2304.08503/) (2023)
- [ScaleNet: Searching for the Model to Scale](https://mlsanity.com/paper/2207.07267/) (2022)
- [Pareto-aware Neural Architecture Generation for Diverse Computational Budgets](https://mlsanity.com/paper/2210.07634/) (2022)
- [Transfer-Once-For-All: AI Model Optimization for Edge](https://mlsanity.com/paper/2303.15485/) (2023)
- [Efficient Stitchable Task Adaptation](https://mlsanity.com/paper/2311.17352/) (2023)
- [Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets](https://mlsanity.com/paper/2103.00219/) (2021)
- [Accelerating Multi-Objective Neural Architecture Search by Random-Weight Evaluation](https://mlsanity.com/paper/2110.05242/) (2021)
- [Towards Automated Error Analysis: Learning to Characterize Errors](https://mlsanity.com/paper/2201.05017/) (2022)

## Abstract (arXiv)

Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Existing NAS approaches require one complete search for each deployment specification of hardware or objective. This is a computationally impractical endeavor given the potentially large number of application scenarios. In this paper, we propose Neural Architecture Transfer (NAT) to overcome this limitation. NAT is designed to efficiently generate task-specific custom models that are competitive under multiple conflicting objectives. To realize this goal we learn task-specific supernets from which specialized subnets can be sampled without any additional training. The key to our approach is an integrated online transfer learning and many-objective evolutionary search procedure. A pre-trained supernet is iteratively adapted while simultaneously searching for task-specific subnets. We demonstrate the efficacy of NAT on 11 benchmark image classification tasks ranging from large-scale multi-class to small-scale fine-grained datasets. In all cases, including ImageNet, NATNets improve upon the state-of-the-art under mobile settings (≤ 600M Multiply-Adds). Surprisingly, small-scale fine-grained datasets benefit the most from NAT. At the same time, the architecture search and transfer is orders of magnitude more efficient than existing NAS methods. Overall, the experimental evaluation indicates that, across diverse image classification tasks and computational objectives, NAT is an appreciably more effective alternative to conventional transfer learning of fine-tuning weights of an existing network architecture learned on standard datasets. Code is available at https://github.com/human-analysis/neural-architecture-transfer