mlsanityData to 7 Oct 2026

Neural Functional Transformers

arXiv 2305.13546 ↗First posted 22 May 2023cs.LG cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

The recent success of neural networks as implicit representation of data has driven growing interest in neural functionals: models that can process other neural networks as input by operating directly over their weight spaces. Nevertheless, constructing expressive and efficient neural functional architectures that can handle high-dimensional weight-space objects remains challenging. This paper uses the attention mechanism to define a novel set of permutation equivariant weight-space layers and composes them into deep equivariant models called neural functional Transformers (NFTs). NFTs respect weight-space permutation symmetries while incorporating the advantages of attention, which have exhibited remarkable success across multiple domains. In experiments processing the weights of feedforward MLPs and CNNs, we find that NFTs match or exceed the performance of prior weight-space methods. We also leverage NFTs to develop Inr2Array, a novel method for computing permutation invariant latent representations from the weights of implicit neural representations (INRs). Our proposed method improves INR classification accuracy by up to +17% over existing methods. We provide an implementation of our layers at https://github.com/AllanYangZhou/nfn.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

38 of 46 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  6. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  7. 2017Deep Setscited by 1,394
  8. 2016Group Equivariant Convolutional Networkscited by 1,154
  9. 2016HyperNetworkscited by 1,003
  10. 2016Learning to learn by gradient descent by gradient descentcited by 616
  11. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  12. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
26 more
  1. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  2. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  3. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  4. 2016Learning to Optimizecited by 195
  5. 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
  6. 2017Equivariance Through Parameter-Sharingcited by 139
  7. 2018Graph HyperNetworks for Neural Architecture Searchcited by 131
  8. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
  9. 2020Predicting Neural Network Accuracy from Weightscited by 107
  10. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  11. 2020On Learning Sets of Symmetric Elementscited by 97
  12. 2021Generative Models as Distributions of Functionscited by 89
  13. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  14. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  15. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  16. 2023Permutation Equivariant Neural Functionalscited by 69
  17. 2018Deep Models of Interactions Across Setscited by 68
  18. 2017Bayesian Hypernetworkscited by 64
  19. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
  20. 2023Deep Learning on Implicit Neural Representations of Shapescited by 62
  21. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  22. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  23. 2023Spatial Functa: Scaling Functa to ImageNet Classification and Generationcited by 55
  24. 2020The general theory of permutation equivarant neural networks and higher order graph variational encoderscited by 28
  25. 2019Towards Task and Architecture-Independent Generalization Gap Predictorscited by 18
  26. 2019A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 14; 2025: 21; 2026: 9.

50 papers.

Sort:

By area:

  1. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  2. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  3. 2023Equivariant Deep Weight Space Alignmentcited by 34
  4. 2025Symmetry in Neural Network Parameter Spacescited by 31
  5. 2024Universal Neural Functionalscited by 25
  6. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  7. 2024Improved Generalization of Weight Space Networks via Augmentationscited by 24
  8. 2024Scale Equivariant Graph Metanetworkscited by 23
  9. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  10. 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
  11. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  12. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
38 more
  1. 2024Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenescited by 12
  2. 2023Set-based Neural Network Encoding Without Weight Tyingcited by 9
  3. 2023Neural Processing of Tri-Plane Hybrid Neural Fieldscited by 8
  4. 2024Learning on Model Weights using Tree Expertscited by 7
  5. 2025End-to-End Implicit Neural Representations for Classificationcited by 7
  6. 2024Connecting NeRFs, Images, and Textcited by 6
  7. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  8. 2023Deep Learning on Object-centric 3D Neural Fieldscited by 5
  9. 2024Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guaranteescited by 5
  10. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
  11. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  12. 2025Weight-Space Linear Recurrent Neural Networkscited by 5
  13. 2025Learning Model Representations Using Publicly Available Model Hubscited by 4
  14. 2025ARC: Anchored Representation Clouds for High-Resolution INR Classificationcited by 3
  15. 2023Data Augmentations in Deep Weight Spacescited by 2
  16. 2024Revisiting Multi-Permutation Equivariance through the Lens of Irreducible Representationscited by 2
  17. 2025Few-shot Implicit Function Generation via Equivariancecited by 2
  18. 2025Neural Functions for Learning Periodic Signalcited by 2
  19. 2025Meta-learning Structure-Preserving Dynamicscited by 2
  20. 2025Topology Guidance: Controlling the Outputs of Generative Models via Vector Field Topologycited by 1
  21. 2025Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruningcited by 1
  22. 2025Beyond Pixels: Medical Image Quality Assessment with Implicit Neural Representationscited by 1
  23. 2025Weight Space Representation Learning via Neural Field Adaptationcited by 1
  24. 2026Ensuring Semantics in Weights of Implicit Neural Representations through the Implicit Function Theoremcited by 1
  25. 2026Quasi-Equivariant Metanetworkscited by 1
  26. 2026Dynamic Neural Graph Encoding of Inference Processes in Deep Weight Spacecited by 1
  27. 2024Wolf2Pack: The AutoFusion Framework for Dynamic Parameter Fusioncited by 0
  28. 2025Weight Space Representation Learning on Diverse NeRF Architecturescited by 0
  29. 2025Adversarial Attacks in Weight-Space Classifierscited by 0
  30. 2025Can We Optimize Deep RL Policy Weights as Trajectory Modeling?cited by 0
  31. 2025Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Trainingcited by 0
  32. 2025Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliancecited by 0
  33. 2026Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglementcited by 0
  34. 2026Is Class Signal Clustered or Routed in Task-Induced Implicit Neural Representation Weight Spaces?cited by 0
  35. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  36. 2026Weight-Space Mixture-of-Experts for Implicit Neural Representation Classificationcited by 0
  37. 2026Finite Probes Suffice: Identifiability and Universality for Weight-Space Learningcited by 0
  38. 2026CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformationscited by 0