# SMASH: One-Shot Model Architecture Search through HyperNetworks

arXiv: https://arxiv.org/abs/1708.05344 · HTML: https://mlsanity.com/paper/1708.05344/

## Key facts

- Cited by 202 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2017.

## Built on by (most cited first, 30 of 202)

- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective](https://mlsanity.com/paper/2102.11535/) (2021)
- [Prompting Decision Transformer for Few-Shot Policy Generalization](https://mlsanity.com/paper/2206.13499/) (2022)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [HyperMorph: Amortized Hyperparameter Learning for Image Registration](https://mlsanity.com/paper/2101.01035/) (2021)
- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.08566/) (2023)
- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning](https://mlsanity.com/paper/2206.03996/) (2022)
- [Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion](https://mlsanity.com/paper/2103.11832/) (2021)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [Neural Network Diffusion](https://mlsanity.com/paper/2402.13144/) (2024)
- [Searching the Search Space of Vision Transformer](https://mlsanity.com/paper/2111.14725/) (2021)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [Learning the Effect of Registration Hyperparameters with HyperMorph](https://mlsanity.com/paper/2203.16680/) (2022)
- [DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalization](https://mlsanity.com/paper/2209.05227/) (2022)
- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [Neural Architecture Search with Random Labels](https://mlsanity.com/paper/2101.11834/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models](https://mlsanity.com/paper/2107.13686/) (2021)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models](https://mlsanity.com/paper/2312.12487/) (2023)
- [Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?](https://mlsanity.com/paper/2303.04143/) (2023)
- [Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization](https://mlsanity.com/paper/2402.12550/) (2024)
- [Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search](https://mlsanity.com/paper/2101.11342/) (2021)

## Abstract (arXiv)

Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. Our code is available at https://github.com/ajbrock/SMASH