# Self-supervised Representation Learning for Evolutionary Neural Architecture Search

arXiv: https://arxiv.org/abs/2011.00186 · HTML: https://mlsanity.com/paper/2011.00186/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [CATE: Computation-aware Neural Architecture Encoding with Transformers](https://mlsanity.com/paper/2102.07108/) (2021)
- [Contrastive Embeddings for Neural Architectures](https://mlsanity.com/paper/2102.04208/) (2021)
- [Anomaly Detection in IR Images of PV Modules using Supervised Contrastive Learning](https://mlsanity.com/paper/2112.02922/) (2021)
- [DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning](https://mlsanity.com/paper/2302.13020/) (2023)
- [Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into Operation Importance](https://mlsanity.com/paper/2303.16938/) (2023)
- [Evolutionary Neural Architecture Search with Dual Contrastive Learning](https://mlsanity.com/paper/2512.20112/) (2025)
- [Explaining Deep Convolutional Neural Networks for Image Classification by Evolving Local Interpretable Model-agnostic Explanations](https://mlsanity.com/paper/2211.15143/) (2022)
- [Towards evolution of Deep Neural Networks through contrastive Self-Supervised learning](https://mlsanity.com/paper/2406.14525/) (2024)

## Abstract (arXiv)

Recently proposed neural architecture search (NAS) algorithms adopt neural predictors to accelerate the architecture search. The capability of neural predictors to accurately predict the performance metrics of neural architecture is critical to NAS, and the acquisition of training datasets for neural predictors is time-consuming. How to obtain a neural predictor with high prediction accuracy using a small amount of training data is a central problem to neural predictor-based NAS. Here, we firstly design a new architecture encoding scheme that overcomes the drawbacks of existing vector-based architecture encoding schemes to calculate the graph edit distance of neural architectures. To enhance the predictive performance of neural predictors, we devise two self-supervised learning methods from different perspectives to pre-train the architecture embedding part of neural predictors to generate a meaningful representation of neural architectures. The first one is to train a carefully designed two branch graph neural network model to predict the graph edit distance of two input neural architectures. The second method is inspired by the prevalently contrastive learning, and we present a new contrastive learning algorithm that utilizes a central feature vector as a proxy to contrast positive pairs against negative pairs. Experimental results illustrate that the pre-trained neural predictors can achieve comparable or superior performance compared with their supervised counterparts with several times less training samples. We achieve state-of-the-art performance on the NASBench-101 and NASBench201 benchmarks when integrating the pre-trained neural predictors with an evolutionary NAS algorithm.