# Generating Neural Networks with Neural Networks

arXiv: https://arxiv.org/abs/1801.01952 · HTML: https://mlsanity.com/paper/1801.01952/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2018.

## Built on by (most cited first, 10 of 10)

- [Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights](https://mlsanity.com/paper/2209.14733/) (2022)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [HyperFast: Instant Classification for Tabular Data](https://mlsanity.com/paper/2402.14335/) (2024)
- [NeRN -- Learning Neural Representations for Neural Networks](https://mlsanity.com/paper/2212.13554/) (2022)
- [Hyper-Representations for Pre-Training and Transfer Learning](https://mlsanity.com/paper/2207.10951/) (2022)
- [Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction](https://mlsanity.com/paper/2503.17138/) (2025)
- [Unleash Graph Neural Networks from Heavy Tuning](https://mlsanity.com/paper/2405.12521/) (2024)
- [Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning](https://mlsanity.com/paper/2410.05697/) (2024)
- [Hyper-Representations: Learning from Populations of Neural Networks](https://mlsanity.com/paper/2410.05107/) (2024)
- [SBS: Enhancing Parameter-Efficiency of Neural Representations for Neural Networks via Spectral Bias Suppression](https://mlsanity.com/paper/2509.07373/) (2025)

## Abstract (arXiv)

Hypernetworks are neural networks that generate weights for another neural network. We formulate the hypernetwork training objective as a compromise between accuracy and diversity, where the diversity takes into account trivial symmetry transformations of the target network. We explain how this simple formulation generalizes variational inference. We use multi-layered perceptrons to form the mapping from the low dimensional input random vector to the high dimensional weight space, and demonstrate how to reduce the number of parameters in this mapping by parameter sharing. We perform experiments and show that the generated weights are diverse and lie on a non-trivial manifold.