# Exploring the Loss Landscape in Neural Architecture Search

arXiv: https://arxiv.org/abs/2005.02960 · HTML: https://mlsanity.com/paper/2005.02960/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2020.

## Built on by (most cited first, 20 of 20)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy](https://mlsanity.com/paper/2201.13396/) (2022)
- [Learning Where To Look -- Generative NAS is Surprisingly Efficient](https://mlsanity.com/paper/2203.08734/) (2022)
- [Neural Architecture Design and Robustness: A Dataset](https://mlsanity.com/paper/2306.06712/) (2023)
- [Multi-conditioned Graph Diffusion for Neural Architecture Search](https://mlsanity.com/paper/2403.06020/) (2024)
- [A Benchmark Study on Calibration](https://mlsanity.com/paper/2308.11838/) (2023)
- [Robust NAS under adversarial training: benchmark, theory, and beyond](https://mlsanity.com/paper/2403.13134/) (2024)
- [Structural Pruning of Pre-trained Language Models via Neural Architecture Search](https://mlsanity.com/paper/2405.02267/) (2024)
- [Scalable Reinforcement Learning-based Neural Architecture Search](https://mlsanity.com/paper/2410.01431/) (2024)
- [Hyperparameter Optimization via Interacting with Probabilistic Circuits](https://mlsanity.com/paper/2505.17804/) (2025)
- [Tackling Neural Architecture Search With Quality Diversity Optimization](https://mlsanity.com/paper/2208.00204/) (2022)
- [Automated Dominative Subspace Mining for Efficient Neural Architecture Search](https://mlsanity.com/paper/2210.17180/) (2022)
- [Shortest Edit Path Crossover: A Theory-driven Solution to the Permutation Problem in Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2210.14016/) (2022)
- [Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Search](https://mlsanity.com/paper/2503.10404/) (2025)
- [Global optimization of graph acquisition functions for neural architecture search](https://mlsanity.com/paper/2505.23640/) (2025)
- [Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvature](https://mlsanity.com/paper/2508.12977/) (2025)
- [Evolutionary Neural Architecture Search with Dual Contrastive Learning](https://mlsanity.com/paper/2512.20112/) (2025)
- [HiveNAS: Neural Architecture Search using Artificial Bee Colony Optimization](https://mlsanity.com/paper/2211.10250/) (2022)
- [OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search](https://mlsanity.com/paper/2509.05656/) (2025)
- [Layered Quantum Architecture Search for 3D Point Cloud Classification](https://mlsanity.com/paper/2603.20024/) (2026)

## Abstract (arXiv)

Neural architecture search (NAS) has seen a steep rise in interest over the last few years. Many algorithms for NAS consist of searching through a space of architectures by iteratively choosing an architecture, evaluating its performance by training it, and using all prior evaluations to come up with the next choice. The evaluation step is noisy - the final accuracy varies based on the random initialization of the weights. Prior work has focused on devising new search algorithms to handle this noise, rather than quantifying or understanding the level of noise in architecture evaluations. In this work, we show that (1) the simplest hill-climbing algorithm is a powerful baseline for NAS, and (2), when the noise in popular NAS benchmark datasets is reduced to a minimum, hill-climbing to outperforms many popular state-of-the-art algorithms. We further back up this observation by showing that the number of local minima is substantially reduced as the noise decreases, and by giving a theoretical characterization of the performance of local search in NAS. Based on our findings, for NAS research we suggest (1) using local search as a baseline, and (2) denoising the training pipeline when possible.