# Training and Generating Neural Networks in Compressed Weight Space

arXiv: https://arxiv.org/abs/2112.15545 · HTML: https://mlsanity.com/paper/2112.15545/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 31 Dec 2021.

## Builds on (most cited first, 9 of 9)

- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016)
- [Predicting Neural Network Accuracy from Weights](https://mlsanity.com/paper/2002.11448/) (2020)
- [An Analysis of Neural Language Modeling at Multiple Scales](https://mlsanity.com/paper/1803.08240/) (2018)
- [Policy Evaluation Networks](https://mlsanity.com/paper/2002.11833/) (2020)
- [Parameter-Based Value Functions](https://mlsanity.com/paper/2006.09226/) (2020)
- [Surprisal-Driven Feedback in Recurrent Networks](https://mlsanity.com/paper/1608.06027/) (2016)

## Built on by (most cited first, 4 of 4)

- [Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules](https://mlsanity.com/paper/2206.01649/) (2022)
- [Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules](https://mlsanity.com/paper/2210.06184/) (2022)
- [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025)
- [Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps](https://mlsanity.com/paper/2506.16787/) (2025)

## Abstract (arXiv)

The inputs and/or outputs of some neural nets are weight matrices of other neural nets. Indirect encodings or end-to-end compression of weight matrices could help to scale such approaches. Our goal is to open a discussion on this topic, starting with recurrent neural networks for character-level language modelling whose weight matrices are encoded by the discrete cosine transform. Our fast weight version thereof uses a recurrent neural network to parameterise the compressed weights. We present experimental results on the enwik8 dataset.