# Graph HyperNetworks for Neural Architecture Search

arXiv: https://arxiv.org/abs/1810.05749 · HTML: https://mlsanity.com/paper/1810.05749/

## Key facts

- Cited by 132 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2018.

## Built on by (most cited first, 30 of 132)

- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [ATT3D: Amortized Text-to-3D Object Synthesis](https://mlsanity.com/paper/2306.07349/) (2023)
- [HyperInverter: Improving StyleGAN Inversion via Hypernetwork](https://mlsanity.com/paper/2112.00719/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [Permutation Equivariant Neural Functionals](https://mlsanity.com/paper/2302.14040/) (2023)
- [Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights](https://mlsanity.com/paper/2209.14733/) (2022)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [Neural Functional Transformers](https://mlsanity.com/paper/2305.13546/) (2023)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [Automated Machine Learning on Graphs: A Survey](https://mlsanity.com/paper/2103.00742/) (2021)
- [Model Zoos: A Dataset of Diverse Populations of Neural Network Models](https://mlsanity.com/paper/2209.14764/) (2022)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalization](https://mlsanity.com/paper/2209.05227/) (2022)
- [Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models](https://mlsanity.com/paper/2508.09968/) (2025)
- [Contrastive Neural Architecture Search with Neural Architecture Comparators](https://mlsanity.com/paper/2103.05471/) (2021)
- [Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?](https://mlsanity.com/paper/2303.04143/) (2023)
- [SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass](https://mlsanity.com/paper/2602.06358/) (2026)
- [HyperTime: Implicit Neural Representation for Time Series](https://mlsanity.com/paper/2208.05836/) (2022)
- [Transformers over Directed Acyclic Graphs](https://mlsanity.com/paper/2210.13148/) (2022)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis](https://mlsanity.com/paper/2210.13300/) (2022)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2307.01998/) (2023)
- [HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models](https://mlsanity.com/paper/2403.13447/) (2024)
- [Functional Bilevel Optimization for Machine Learning](https://mlsanity.com/paper/2403.20233/) (2024)

## Abstract (arXiv)

Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -- they can search nearly 10 times faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.