# Transferrable Surrogates in Expressive Neural Architecture Search Spaces

arXiv: https://arxiv.org/abs/2504.12971 · HTML: https://mlsanity.com/paper/2504.12971/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Builds on (most cited first, 30 of 33)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Neural Architecture Search without Training](https://mlsanity.com/paper/2006.04647/) (2020)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [BRP-NAS: Prediction-based NAS using GCNs](https://mlsanity.com/paper/2007.08668/) (2020)
- [How Powerful are Performance Predictors in Neural Architecture Search?](https://mlsanity.com/paper/2104.01177/) (2021)
- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks](https://mlsanity.com/paper/2008.09777/) (2020)
- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?](https://mlsanity.com/paper/2006.06936/) (2020)
- [Evaluating Efficient Performance Estimators of Neural Architectures](https://mlsanity.com/paper/2008.03064/) (2020)
- [Searching the Search Space of Vision Transformer](https://mlsanity.com/paper/2111.14725/) (2021)
- [NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies](https://mlsanity.com/paper/2210.03230/) (2022)
- [LLM Performance Predictors are good initializers for Architecture Search](https://mlsanity.com/paper/2310.16712/) (2023)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [Design Principle Transfer in Neural Architecture Search via Large Language Models](https://mlsanity.com/paper/2408.11330/) (2024)
- [Large Language Model-driven Meta-structure Discovery in Heterogeneous Information Network](https://mlsanity.com/paper/2402.11518/) (2024)
- [Surprisingly Strong Performance Prediction with Neural Graph Features](https://mlsanity.com/paper/2404.16551/) (2024)
- [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://mlsanity.com/paper/2211.01842/) (2022)
- [Encodings for Prediction-based Neural Architecture Search](https://mlsanity.com/paper/2403.02484/) (2024)

## Built on by (most cited first, 5 of 5)

- [AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search](https://mlsanity.com/paper/2506.06017/) (2025)
- [Regression Language Models for Code](https://mlsanity.com/paper/2509.26476/) (2025)
- [ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures](https://mlsanity.com/paper/2510.04938/) (2025)
- [A Modality-Aware Cooperative Co-Evolutionary Framework for Multimodal Graph Neural Architecture Search](https://mlsanity.com/paper/2510.07325/) (2025)
- [Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Search](https://mlsanity.com/paper/2605.15649/) (2026)

## Abstract (arXiv)

Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups.