# HyperNetworks

arXiv: https://arxiv.org/abs/1609.09106 · HTML: https://mlsanity.com/paper/1609.09106/

## Key facts

- Cited by 1,027 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2016.

## Built on by (most cited first, 30 of 1027)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2307.06949/) (2023)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023)
- [UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild](https://mlsanity.com/paper/2305.11147/) (2023)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up](https://mlsanity.com/paper/2102.07074/) (2021)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Modulated Periodic Activations for Generalizable Local Functional Representations](https://mlsanity.com/paper/2104.03960/) (2021)
- [Local Texture Estimator for Implicit Representation Function](https://mlsanity.com/paper/2111.08918/) (2021)
- [EpiGRAF: Rethinking training of 3D GANs](https://mlsanity.com/paper/2206.10535/) (2022)
- [LambdaNetworks: Modeling Long-Range Interactions Without Attention](https://mlsanity.com/paper/2102.08602/) (2021)
- [General-Purpose In-Context Learning by Meta-Learning Transformers](https://mlsanity.com/paper/2212.04458/) (2022)
- [Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation](https://mlsanity.com/paper/2403.15356/) (2024)
- [Stylizing 3D Scene via Implicit Representation and HyperNetwork](https://mlsanity.com/paper/2105.13016/) (2021)

## Abstract (arXiv)

This work explores hypernetworks: an approach of using a one network, also known as a hypernetwork, to generate the weights for another network. Hypernetworks provide an abstraction that is similar to what is found in nature: the relationship between a genotype - the hypernetwork - and a phenotype - the main network. Though they are also reminiscent of HyperNEAT in evolution, our hypernetworks are trained end-to-end with backpropagation and thus are usually faster. The focus of this work is to make hypernetworks useful for deep convolutional networks and long recurrent networks, where hypernetworks can be viewed as relaxed form of weight-sharing across layers. Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve near state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks. Our results also show that hypernetworks applied to convolutional networks still achieve respectable results for image recognition tasks compared to state-of-the-art baseline models while requiring fewer learnable parameters.