mlsanityData to 14 Oct 2026

Transferrable Surrogates in Expressive Neural Architecture Search Spaces

arXiv 2504.12971 ↗First posted 17 Apr 2025cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

33 of 44 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2017Graph Attention Networkscited by 8,923
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  6. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  7. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  8. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  9. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  10. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  11. 2020Neural Architecture Search without Trainingcited by 244
  12. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
21 more
  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  2. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  3. 2020BRP-NAS: Prediction-based NAS using GCNscited by 111
  4. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 90
  5. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 87
  6. 2020Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarkscited by 67
  7. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  8. 2020Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?cited by 59
  9. 2020Evaluating Efficient Performance Estimators of Neural Architecturescited by 55
  10. 2021Searching the Search Space of Vision Transformercited by 41
  11. 2022NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxiescited by 36
  12. 2023LLM Performance Predictors are good initializers for Architecture Searchcited by 29
  13. 2021NAS-Bench-x11 and the Power of Learning Curvescited by 20
  14. 2024Design Principle Transfer in Neural Architecture Search via Large Language Modelscited by 17
  15. 2024Large Language Model-driven Meta-structure Discovery in Heterogeneous Information Networkcited by 16
  16. 2024Surprisingly Strong Performance Prediction with Neural Graph Featurescited by 11
  17. 2022Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammarscited by 9
  18. 2024Encodings for Prediction-based Neural Architecture Searchcited by 9
  19. 2024STAR: Synthesis of Tailored Architecturescited by 7
  20. 2024Insights from the Use of Previously Unseen Neural Architecture Search Datasetscited by 5
  21. 2024einspace: Searching for Neural Architectures from Fundamental Operationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Searchcited by 15
  2. 2025Regression Language Models for Codecited by 11
  3. 2025ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architecturescited by 1
  4. 2025A Modality-Aware Cooperative Co-Evolutionary Framework for Multimodal Graph Neural Architecture Searchcited by 0
  5. 2026Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Searchcited by 0