# Permutation Equivariant Neural Functionals

arXiv: https://arxiv.org/abs/2302.14040 · HTML: https://mlsanity.com/paper/2302.14040/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 73 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups](https://mlsanity.com/paper/2104.09459/) (2021)
- [Editable Neural Networks](https://mlsanity.com/paper/2004.00345/) (2020)
- [Equivariance Through Parameter-Sharing](https://mlsanity.com/paper/1702.08389/) (2017)

## Built on by (most cited first, 30 of 70)

- [Graph Neural Networks for Learning Equivariant Representations of Neural Networks](https://mlsanity.com/paper/2403.12143/) (2024)
- [Graph Metanetworks for Processing Diverse Neural Architectures](https://mlsanity.com/paper/2312.04501/) (2023)
- [Neural Functional Transformers](https://mlsanity.com/paper/2305.13546/) (2023)
- [Neural Redshift: Random Networks are not Random Functions](https://mlsanity.com/paper/2403.02241/) (2024)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Equivariant Deep Weight Space Alignment](https://mlsanity.com/paper/2310.13397/) (2023)
- [Universal Neural Functionals](https://mlsanity.com/paper/2402.05232/) (2024)
- [We Should Chart an Atlas of All the World's Models](https://mlsanity.com/paper/2503.10633/) (2025)
- [SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass](https://mlsanity.com/paper/2602.06358/) (2026)
- [Improved Generalization of Weight Space Networks via Augmentations](https://mlsanity.com/paper/2402.04081/) (2024)
- [Scale Equivariant Graph Metanetworks](https://mlsanity.com/paper/2406.10685/) (2024)
- [Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models](https://mlsanity.com/paper/2410.04207/) (2024)
- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [Generating Behaviorally Diverse Policies with Latent Diffusion Models](https://mlsanity.com/paper/2305.18738/) (2023)
- [Monomial Matrix Group Equivariant Neural Functional Networks](https://mlsanity.com/paper/2409.11697/) (2024)
- [Deep Linear Probe Generators for Weight Space Learning](https://mlsanity.com/paper/2410.10811/) (2024)
- [Learning Useful Representations of Recurrent Neural Network Weight Matrices](https://mlsanity.com/paper/2403.11998/) (2024)
- [How to Train Neural Field Representations: A Comprehensive Study and Benchmark](https://mlsanity.com/paper/2312.10531/) (2023)
- [Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning](https://mlsanity.com/paper/2506.10973/) (2025)
- [Robot Fleet Learning via Policy Merging](https://mlsanity.com/paper/2310.01362/) (2023)
- [Unsupervised Model Tree Heritage Recovery](https://mlsanity.com/paper/2405.18432/) (2024)
- [From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches](https://mlsanity.com/paper/2503.08998/) (2025)
- [Set-based Neural Network Encoding Without Weight Tying](https://mlsanity.com/paper/2305.16625/) (2023)
- [Neural Processing of Tri-Plane Hybrid Neural Fields](https://mlsanity.com/paper/2310.01140/) (2023)
- [Learning on Model Weights using Tree Experts](https://mlsanity.com/paper/2410.13569/) (2024)
- [End-to-End Implicit Neural Representations for Classification](https://mlsanity.com/paper/2503.18123/) (2025)
- [GradMetaNet: An Equivariant Architecture for Learning on Gradients](https://mlsanity.com/paper/2507.01649/) (2025)
- [Connecting NeRFs, Images, and Text](https://mlsanity.com/paper/2404.07993/) (2024)
- [LLaNA: Large Language and NeRF Assistant](https://mlsanity.com/paper/2406.11840/) (2024)
- [Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction](https://mlsanity.com/paper/2503.17138/) (2025)

## Abstract (arXiv)

This work studies the design of neural networks that can process the weights or gradients of other neural networks, which we refer to as neural functional networks (NFNs). Despite a wide range of potential applications, including learned optimization, processing implicit neural representations, network editing, and policy evaluation, there are few unifying principles for designing effective architectures that process the weights of other networks. We approach the design of neural functionals through the lens of symmetry, in particular by focusing on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. We introduce a framework for building permutation equivariant neural functionals, whose architectures encode these symmetries as an inductive bias. The key building blocks of this framework are NF-Layers (neural functional layers) that we constrain to be permutation equivariant through an appropriate parameter sharing scheme. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks that require processing the weights of MLPs and CNNs, such as predicting classifier generalization, producing "winning ticket" sparsity masks for initializations, and classifying or editing implicit neural representations (INRs). In addition, we provide code for our models and experiments at https://github.com/AllanYangZhou/nfn.