# Understanding Neural Architecture Search Techniques

arXiv: https://arxiv.org/abs/1904.00438 · HTML: https://mlsanity.com/paper/1904.00438/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2019.

## Built on by (most cited first, 12 of 12)

- [Complex Momentum for Optimization in Games](https://mlsanity.com/paper/2102.08431/) (2021)
- [Normalizing Flows for Conformal Regression](https://mlsanity.com/paper/2406.03346/) (2024)
- [Lyapunov Exponents for Diversity in Differentiable Games](https://mlsanity.com/paper/2112.14570/) (2021)
- [Task Selection for AutoML System Evaluation](https://mlsanity.com/paper/2208.12754/) (2022)
- [JacNet: Learning Functions with Structured Jacobians](https://mlsanity.com/paper/2408.13237/) (2024)
- [Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond](https://mlsanity.com/paper/2505.04621/) (2025)
- [Learning Interpretable Models Through Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2112.08645/) (2021)
- [Improving Hyperparameter Optimization with Checkpointed Model Weights](https://mlsanity.com/paper/2406.18630/) (2024)
- [FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer](https://mlsanity.com/paper/2311.03912/) (2023)
- [Efficient Neural Architecture Search with Performance Prediction](https://mlsanity.com/paper/2108.01854/) (2021)
- [Auto Machine Learning for Medical Image Analysis by Unifying the Search on Data Augmentation and Neural Architecture](https://mlsanity.com/paper/2207.10351/) (2022)
- [Deep Active Learning with Structured Neural Depth Search](https://mlsanity.com/paper/2306.02808/) (2023)

## Abstract (arXiv)

Automatic methods for generating state-of-the-art neural network architectures without human experts have generated significant attention recently. This is because of the potential to remove human experts from the design loop which can reduce costs and decrease time to model deployment. Neural architecture search (NAS) techniques have improved significantly in their computational efficiency since the original NAS was proposed. This reduction in computation is enabled via weight sharing such as in Efficient Neural Architecture Search (ENAS). However, recently a body of work confirms our discovery that ENAS does not do significantly better than random search with weight sharing, contradicting the initial claims of the authors. We provide an explanation for this phenomenon by investigating the interpretability of the ENAS controller's hidden state. We find models sampled from identical controller hidden states have no correlation with various graph similarity metrics, so no notion of structural similarity is learned. This failure mode implies the RNN controller does not condition on past architecture choices. Lastly, we propose a solution to this failure mode by forcing the controller's hidden state to encode pasts decisions by training it with a memory buffer of previously sampled architectures. Doing this improves hidden state interpretability by increasing the correlation between controller hidden states and graph similarity metrics.