# Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning

arXiv: https://arxiv.org/abs/2103.08157 · HTML: https://mlsanity.com/paper/2103.08157/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 46 of its 55 references that are in the map.
- First posted on arXiv 15 Mar 2021.

## Builds on (most cited first, 30 of 46)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation](https://mlsanity.com/paper/1901.02985/) (2019)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [Hierarchical Representations for Efficient Architecture Search](https://mlsanity.com/paper/1711.00436/) (2017)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)

## Abstract (arXiv)

Neural architecture search (NAS) has fostered various fields of machine learning. Despite its prominent dedications, many have criticized the intrinsic limitations of high computational cost. We aim to ameliorate this by proposing a pretraining scheme that can be generally applied to controller-based NAS. Our method, locality-based self-supervised classification task, leverages the structural similarity of network architectures to obtain good architecture representations. We incorporate our method into neural architecture optimization (NAO) to analyze the pretrained embeddings and its effectiveness and highlight that adding metric learning loss brings a favorable impact on NAS. Our code is available at urlhttps://github.com/Multi-Objective-NAS/self-supervised-nas.