mlsanityData to 7 Oct 2026

Permutation Equivariant Neural Functionals

arXiv 2302.14040 ↗First posted 27 Feb 2023cs.LG cs.AIcited by 69

Not on the map. Why?

Abstract (arXiv)

This work studies the design of neural networks that can process the weights or gradients of other neural networks, which we refer to as neural functional networks (NFNs). Despite a wide range of potential applications, including learned optimization, processing implicit neural representations, network editing, and policy evaluation, there are few unifying principles for designing effective architectures that process the weights of other networks. We approach the design of neural functionals through the lens of symmetry, in particular by focusing on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. We introduce a framework for building permutation equivariant neural functionals, whose architectures encode these symmetries as an inductive bias. The key building blocks of this framework are NF-Layers (neural functional layers) that we constrain to be permutation equivariant through an appropriate parameter sharing scheme. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks that require processing the weights of MLPs and CNNs, such as predicting classifier generalization, producing "winning ticket" sparsity masks for initializations, and classifying or editing implicit neural representations (INRs). In addition, we provide code for our models and experiments at https://github.com/AllanYangZhou/nfn.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 51 of its 73 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

51 of 73 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  8. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  12. 2017Deep Setscited by 1,394
38 more
  1. 2016Group Equivariant Convolutional Networkscited by 1,154
  2. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  3. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  4. 2016HyperNetworkscited by 1,003
  5. 2016Learning to learn by gradient descent by gradient descentcited by 616
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  7. 2021Editing Factual Knowledge in Language Modelscited by 548
  8. 2021Fast Model Editing at Scalecited by 491
  9. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  10. 2018Spherical CNNscited by 452
  11. 2018Invariant and Equivariant Graph Networkscited by 339
  12. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  13. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  14. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  15. 2016Learning to Optimizecited by 195
  16. 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
  17. 2020Editable Neural Networkscited by 154
  18. 2017Equivariance Through Parameter-Sharingcited by 139
  19. 2018Graph HyperNetworks for Neural Architecture Searchcited by 131
  20. 2019PointDAN: A Multi-Scale 3D Domain Adaption Network for Point Cloud Representationcited by 114
  21. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
  22. 2020Predicting Neural Network Accuracy from Weightscited by 107
  23. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  24. 2020On Learning Sets of Symmetric Elementscited by 97
  25. 2021Generative Models as Distributions of Functionscited by 89
  26. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  27. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  28. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  29. 2018Deep Models of Interactions Across Setscited by 68
  30. 2017Bayesian Hypernetworkscited by 64
  31. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  32. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
  33. 2023Deep Learning on Implicit Neural Representations of Shapescited by 62
  34. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  35. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  36. 2020Policy Evaluation Networkscited by 28
  37. 2020The general theory of permutation equivarant neural networks and higher order graph variational encoderscited by 28
  38. 2019Towards Task and Architecture-Independent Generalization Gap Predictorscited by 18

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 20; 2025: 26; 2026: 13.

69 papers.

Sort:

By area:

  1. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  2. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  3. 2023Neural Functional Transformerscited by 50
  4. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  5. 2023Equivariant Deep Weight Space Alignmentcited by 34
  6. 2025Symmetry in Neural Network Parameter Spacescited by 31
  7. 2024Universal Neural Functionalscited by 25
  8. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  9. 2024Improved Generalization of Weight Space Networks via Augmentationscited by 24
  10. 2024Scale Equivariant Graph Metanetworkscited by 23
  11. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 22
  12. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
38 more
  1. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  2. 2023Generating Behaviorally Diverse Policies with Latent Diffusion Modelscited by 18
  3. 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
  4. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
  5. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 13
  6. 2023How to Train Neural Field Representations: A Comprehensive Study and Benchmarkcited by 12
  7. 2023Robot Fleet Learning via Policy Mergingcited by 11
  8. 2024Unsupervised Model Tree Heritage Recoverycited by 11
  9. 2025From Task-Specific Models to Unified Systems: A Review of Model Merging Approachescited by 11
  10. 2025Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learningcited by 11
  11. 2023Set-based Neural Network Encoding Without Weight Tyingcited by 9
  12. 2023Neural Processing of Tri-Plane Hybrid Neural Fieldscited by 8
  13. 2024Learning on Model Weights using Tree Expertscited by 7
  14. 2025End-to-End Implicit Neural Representations for Classificationcited by 7
  15. 2025GradMetaNet: An Equivariant Architecture for Learning on Gradientscited by 7
  16. 2024Connecting NeRFs, Images, and Textcited by 6
  17. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  18. 2023Deep Learning on Object-centric 3D Neural Fieldscited by 5
  19. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
  20. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  21. 2025Weight-Space Linear Recurrent Neural Networkscited by 5
  22. 2025ARC: Anchored Representation Clouds for High-Resolution INR Classificationcited by 3
  23. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  24. 2023Data Augmentations in Deep Weight Spacescited by 2
  25. 2024Current Symmetry Group Equivariant Convolution Frameworks for Representation Learningcited by 2
  26. 2024Revisiting Multi-Permutation Equivariance through the Lens of Irreducible Representationscited by 2
  27. 2025Few-shot Implicit Function Generation via Equivariancecited by 2
  28. 2025Improving Learning to Optimize Using Parameter Symmetriescited by 2
  29. 2024Layer-wise Model Merging for Unsupervised Domain Adaptation in Segmentation Taskscited by 1
  30. 2024CopRA: A Progressive LoRA Training Strategycited by 1
  31. 2025Massively Scaling Explicit Policy-conditioned Value Functionscited by 1
  32. 2025TeleLoRA: Teleporting Model-Specific Alignment Across LLMscited by 1
  33. 2025The Impact of Model Zoo Size and Composition on Weight Space Learningcited by 1
  34. 2025Origin Tracer: A Method for Detecting LoRA Fine-Tuning Origins in LLMscited by 1
  35. 2025Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruningcited by 1
  36. 2025Flows and Diffusions on the Neural Manifoldcited by 1
  37. 2025Weight Space Representation Learning via Neural Field Adaptationcited by 1
  38. 2026Ensuring Semantics in Weights of Implicit Neural Representations through the Implicit Function Theoremcited by 1

50 most cited of 69 All 69 (JSON)