# Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

arXiv: https://arxiv.org/abs/2006.06936 · HTML: https://mlsanity.com/paper/2006.06936/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 30 of 59)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [Neural Architecture Search with Random Labels](https://mlsanity.com/paper/2101.11834/) (2021)
- [Stronger NAS with Weaker Predictors](https://mlsanity.com/paper/2102.10490/) (2021)
- [CATE: Computation-aware Neural Architecture Encoding with Transformers](https://mlsanity.com/paper/2102.07108/) (2021)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [PACE: A Parallelizable Computation Encoder for Directed Acyclic Graphs](https://mlsanity.com/paper/2203.10304/) (2022)
- [On Redundancy and Diversity in Cell-based Neural Architecture Search](https://mlsanity.com/paper/2203.08887/) (2022)
- [Learning Where To Look -- Generative NAS is Surprisingly Efficient](https://mlsanity.com/paper/2203.08734/) (2022)
- [Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift](https://mlsanity.com/paper/2108.09671/) (2021)
- [Multi-conditioned Graph Diffusion for Neural Architecture Search](https://mlsanity.com/paper/2403.06020/) (2024)
- [Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networks](https://mlsanity.com/paper/2409.12255/) (2024)
- [Regression Language Models for Code](https://mlsanity.com/paper/2509.26476/) (2025)
- [Learning Versatile Neural Architectures by Propagating Network Codes](https://mlsanity.com/paper/2103.13253/) (2021)
- [RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving](https://mlsanity.com/paper/2108.08019/) (2021)
- [Encodings for Prediction-based Neural Architecture Search](https://mlsanity.com/paper/2403.02484/) (2024)
- [Task-Adaptive Neural Network Search with Meta-Contrastive Learning](https://mlsanity.com/paper/2103.01495/) (2021)
- [On Latency Predictors for Neural Architecture Search](https://mlsanity.com/paper/2403.02446/) (2024)
- [L2NAS: Learning to Optimize Neural Architectures via Continuous-Action Reinforcement Learning](https://mlsanity.com/paper/2109.12425/) (2021)
- [Graph-based Neural Architecture Search with Operation Embeddings](https://mlsanity.com/paper/2105.04885/) (2021)
- [Interpreting the Mechanism of Synergism for Drug Combinations Using Attention-Based Hierarchical Graph Pooling](https://mlsanity.com/paper/2209.09245/) (2022)
- [ParZC: Parametric Zero-Cost Proxies for Efficient NAS](https://mlsanity.com/paper/2402.02105/) (2024)
- [Unsupervised Graph Neural Architecture Search with Disentangled Self-supervision](https://mlsanity.com/paper/2403.05064/) (2024)
- [CAP: A Context-Aware Neural Predictor for NAS](https://mlsanity.com/paper/2406.02056/) (2024)
- [Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equation](https://mlsanity.com/paper/2412.15554/) (2024)
- [Transferrable Surrogates in Expressive Neural Architecture Search Spaces](https://mlsanity.com/paper/2504.12971/) (2025)
- [MONCAE: Multi-Objective Neuroevolution of Convolutional Autoencoders](https://mlsanity.com/paper/2106.11914/) (2021)
- [Sampling Through the Lens of Sequential Decision Making](https://mlsanity.com/paper/2208.08056/) (2022)
- [Quantum Architecture Search with Unsupervised Representation Learning](https://mlsanity.com/paper/2401.11576/) (2024)

## Abstract (arXiv)

Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels considerably improve the downstream architecture search efficiency. To explain these observations, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together. This helps to map neural architectures with similar performance to the same regions in the latent space and makes the transition of architectures in the latent space relatively smooth, which considerably benefits diverse downstream search strategies.