mlsanityData to 7 Oct 2026

Universal Neural Functionals

arXiv 2402.05232 ↗First posted 7 Feb 2024cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

A challenging problem in many modern machine learning tasks is to process weight-space features, i.e., to transform or extract information from the weights and gradients of a neural network. Recent works have developed promising weight-space models that are equivariant to the permutation symmetries of simple feedforward networks. However, they are not applicable to general architectures, since the permutation symmetries of a weight space can be complicated by recurrence or residual connections. This work proposes an algorithm that automatically constructs permutation equivariant models, which we refer to as universal neural functionals (UNFs), for any weight space. Among other applications, we demonstrate how UNFs can be substituted into existing learned optimizer designs, and find promising improvements over prior methods when optimizing small image classifiers and language models. Our results suggest that learned optimizers can benefit from considering the (symmetry) structure of the weight space they optimize. We open-source our library for constructing UNFs at https://github.com/AllanYangZhou/universal_neural_functional.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

33 of 46 references.

References · 33

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  4. 2016Group Equivariant Convolutional Networkscited by 1,154
  5. 2016HyperNetworkscited by 1,003
  6. 2016Learning to learn by gradient descent by gradient descentcited by 616
  7. 2018Spherical CNNscited by 452
  8. 2018Invariant and Equivariant Graph Networkscited by 339
  9. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  10. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  11. 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
  12. 2017Equivariance Through Parameter-Sharingcited by 139
21 more
  1. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  2. 2017Learned Optimizers that Scale and Generalizecited by 109
  3. 2020Predicting Neural Network Accuracy from Weightscited by 107
  4. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  5. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  6. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  7. 2023Permutation Equivariant Neural Functionalscited by 69
  8. 2018Deep Models of Interactions Across Setscited by 68
  9. 2017Bayesian Hypernetworkscited by 64
  10. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  11. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
  12. 2023Deep Learning on Implicit Neural Representations of Shapescited by 62
  13. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  14. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  15. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  16. 2023Neural Functional Transformerscited by 50
  17. 2020Meta Learning Backpropagation And Improving Itcited by 49
  18. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  19. 2020The general theory of permutation equivarant neural networks and higher order graph variational encoderscited by 28
  20. 2021Introducing Symmetries to Black Box Meta Reinforcement Learningcited by 25
  21. 2019Incidence Networks for Geometric Deep Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 11; 2026: 4.

25 papers.

Sort:

By area:

  1. 2025Symmetry in Neural Network Parameter Spacescited by 31
  2. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  3. 2024Scale Equivariant Graph Metanetworkscited by 23
  4. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  5. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  6. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
  7. 2024Learning on Model Weights using Tree Expertscited by 7
  8. 2025GradMetaNet: An Equivariant Architecture for Learning on Gradientscited by 7
  9. 2024Connecting NeRFs, Images, and Textcited by 6
  10. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  11. 2024Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networkscited by 6
  12. 2025Weight-Space Linear Recurrent Neural Networkscited by 5
13 more
  1. 2025A Model Zoo of Vision Transformerscited by 3
  2. 2024Universal Approximation of Operators with Transformers and Neural Integral Operatorscited by 2
  3. 2025Improving Learning to Optimize Using Parameter Symmetriescited by 2
  4. 2025Massively Scaling Explicit Policy-conditioned Value Functionscited by 1
  5. 2025Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruningcited by 1
  6. 2026A Graph Meta-Network for Learning on Kolmogorov-Arnold Networkscited by 1
  7. 2024Hyper-Representations: Learning from Populations of Neural Networkscited by 0
  8. 2025Adversarial Attacks in Weight-Space Classifierscited by 0
  9. 2025Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Trainingcited by 0
  10. 2025Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalizationcited by 0
  11. 2026Is Class Signal Clustered or Routed in Task-Induced Implicit Neural Representation Weight Spaces?cited by 0
  12. 2026Imprint Reader: From Weight-Update Readout to Behavioral Interventioncited by 0
  13. 2026CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformationscited by 0