# A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks

arXiv: https://arxiv.org/abs/1905.02898 · HTML: https://mlsanity.com/paper/1905.02898/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2019.

## Built on by (most cited first, 12 of 12)

- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [Learning to Learn with Generative Models of Neural Network Checkpoints](https://mlsanity.com/paper/2209.12892/) (2022)
- [Permutation Equivariant Neural Functionals](https://mlsanity.com/paper/2302.14040/) (2023)
- [Neural Functional Transformers](https://mlsanity.com/paper/2305.13546/) (2023)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [Principled Weight Initialization for Hypernetworks](https://mlsanity.com/paper/2312.08399/) (2023)
- [Example-based Hypernetworks for Out-of-Distribution Generalization](https://mlsanity.com/paper/2203.14276/) (2022)
- [Dynamic Inter-treatment Information Sharing for Individualized Treatment Effects Estimation](https://mlsanity.com/paper/2305.15984/) (2023)
- [Robot Fleet Learning via Policy Merging](https://mlsanity.com/paper/2310.01362/) (2023)
- [Individualised Treatment Effects Estimation with Composite Treatments and Composite Outcomes](https://mlsanity.com/paper/2502.08282/) (2025)
- [MinConvNets: A new class of multiplication-less Neural Networks](https://mlsanity.com/paper/2101.09492/) (2021)
- [Universal Hypernetworks for Arbitrary Models](https://mlsanity.com/paper/2604.02215/) (2026)

## Abstract (arXiv)

Recent work on mode connectivity in the loss landscape of deep neural networks has demonstrated that the locus of (sub-)optimal weight vectors lies on continuous paths. In this work, we train a neural network that serves as a hypernetwork, mapping a latent vector into high-performance (low-loss) weight vectors, generalizing recent findings of mode connectivity to higher dimensional manifolds. We formulate the training objective as a compromise between accuracy and diversity, where the diversity takes into account trivial symmetry transformations of the target network. We demonstrate how to reduce the number of parameters in the hypernetwork by parameter sharing. Once learned, the hypernetwork allows for a computationally efficient, ancestral sampling of neural network weights, which we recruit to form large ensembles. The improvement in classification accuracy obtained by this ensembling indicates that the generated manifold extends in dimensions other than directions implied by trivial symmetries. For computational efficiency, we distill an ensemble into a single classifier while retaining generalization.