# Learning Transferable Architectures for Scalable Image Recognition

arXiv: https://arxiv.org/abs/1707.07012 · HTML: https://mlsanity.com/paper/1707.07012/

## Key facts

- Cited by 1,101 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2017.

## Built on by (most cited first, 30 of 1101)

- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021)
- [Coordinate Attention for Efficient Mobile Network Design](https://mlsanity.com/paper/2103.02907/) (2021)
- [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](https://mlsanity.com/paper/2103.12731/) (2021)
- [Parametric Contrastive Learning](https://mlsanity.com/paper/2107.12028/) (2021)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](https://mlsanity.com/paper/2305.07027/) (2023)
- [Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective](https://mlsanity.com/paper/2102.11535/) (2021)
- [VOLO: Vision Outlooker for Visual Recognition](https://mlsanity.com/paper/2106.13112/) (2021)
- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [Revisiting ResNets: Improved Training and Scaling Strategies](https://mlsanity.com/paper/2103.07579/) (2021)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Neural Prompt Search](https://mlsanity.com/paper/2206.04673/) (2022)
- [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges](https://mlsanity.com/paper/2103.04505/) (2021)
- [Involution: Inverting the Inherence of Convolution for Visual Recognition](https://mlsanity.com/paper/2103.06255/) (2021)
- [A Comprehensive Survey of Image Augmentation Techniques for Deep Learning](https://mlsanity.com/paper/2205.01491/) (2022)
- [ResLT: Residual Learning for Long-tailed Recognition](https://mlsanity.com/paper/2101.10633/) (2021)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting](https://mlsanity.com/paper/2304.03307/) (2023)

## Abstract (arXiv)

Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (the "NASNet search space") which enables transferability. In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, named "NASNet architecture". We also introduce a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models. On CIFAR-10 itself, NASNet achieves 2.4% error rate, which is state-of-the-art. On ImageNet, NASNet achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet. Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS - a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than equivalently-sized, state-of-the-art models for mobile platforms. Finally, the learned features by NASNet used with the Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.