# NeRN -- Learning Neural Representations for Neural Networks

arXiv: https://arxiv.org/abs/2212.13554 · HTML: https://mlsanity.com/paper/2212.13554/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 27 Dec 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [COIN: COmpression with Implicit Neural representations](https://mlsanity.com/paper/2103.03123/) (2021)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Knowledge Distillation with Adversarial Samples Supporting Decision Boundary](https://mlsanity.com/paper/1805.05532/) (2018)
- [Hyper-Representations for Pre-Training and Transfer Learning](https://mlsanity.com/paper/2207.10951/) (2022)

## Built on by (most cited first, 14 of 14)

- [Graph Neural Networks for Learning Equivariant Representations of Neural Networks](https://mlsanity.com/paper/2403.12143/) (2024)
- [Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?](https://mlsanity.com/paper/2303.04143/) (2023)
- [We Should Chart an Atlas of All the World's Models](https://mlsanity.com/paper/2503.10633/) (2025)
- [Monomial Matrix Group Equivariant Neural Functional Networks](https://mlsanity.com/paper/2409.11697/) (2024)
- [Deep Linear Probe Generators for Weight Space Learning](https://mlsanity.com/paper/2410.10811/) (2024)
- [Learning on Model Weights using Tree Experts](https://mlsanity.com/paper/2410.13569/) (2024)
- [ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion](https://mlsanity.com/paper/2503.24354/) (2025)
- [Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models](https://mlsanity.com/paper/2304.13718/) (2023)
- [Neural Metamorphosis](https://mlsanity.com/paper/2410.11878/) (2024)
- [A Model Zoo on Phase Transitions in Neural Networks](https://mlsanity.com/paper/2504.18072/) (2025)
- [Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization](https://mlsanity.com/paper/2407.00356/) (2024)
- [RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks](https://mlsanity.com/paper/2411.16870/) (2024)
- [GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shifts](https://mlsanity.com/paper/2502.12195/) (2025)
- [SBS: Enhancing Parameter-Efficiency of Neural Representations for Neural Networks via Spectral Bias Suppression](https://mlsanity.com/paper/2509.07373/) (2025)

## Abstract (arXiv)

Neural Representations have recently been shown to effectively reconstruct a wide range of signals from 3D meshes and shapes to images and videos. We show that, when adapted correctly, neural representations can be used to directly represent the weights of a pre-trained convolutional neural network, resulting in a Neural Representation for Neural Networks (NeRN). Inspired by coordinate inputs of previous neural representation methods, we assign a coordinate to each convolutional kernel in our network based on its position in the architecture, and optimize a predictor network to map coordinates to their corresponding weights. Similarly to the spatial smoothness of visual scenes, we show that incorporating a smoothness constraint over the original network's weights aids NeRN towards a better reconstruction. In addition, since slight perturbations in pre-trained model weights can result in a considerable accuracy loss, we employ techniques from the field of knowledge distillation to stabilize the learning process. We demonstrate the effectiveness of NeRN in reconstructing widely used architectures on CIFAR-10, CIFAR-100, and ImageNet. Finally, we present two applications using NeRN, demonstrating the capabilities of the learned representations.