mlsanityData to 7 Oct 2026

Equivariant Architectures for Learning in Deep Weight Spaces

arXiv 2301.12780 ↗First posted 30 Jan 2023cs.LGcited by 106

Not on the map. Why?

Abstract (arXiv)

Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 55 of its 82 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

55 of 82 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2017Graph Attention Networkscited by 8,854
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  7. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  8. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  9. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  10. 2016Group Equivariant Convolutional Networkscited by 1,154
  11. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  12. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 878
38 more
  1. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  2. 2020Federated Learning with Matched Averagingcited by 564
  3. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  4. 2018Spherical CNNscited by 452
  5. 2019Provably Powerful Graph Networkscited by 385
  6. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  7. 2018Invariant and Equivariant Graph Networkscited by 339
  8. 2016Steerable CNNscited by 298
  9. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  10. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  11. 2017Learning SO(3) Equivariant Representations with Spherical CNNscited by 244
  12. 2019Model Fusion via Optimal Transportcited by 217
  13. 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
  14. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  15. 2019Universal Invariant and Equivariant Graph Neural Networkscited by 182
  16. 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 176
  17. 2019On the Universality of Invariant Networkscited by 151
  18. 2017Equivariance Through Parameter-Sharingcited by 139
  19. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  20. 2021Weisfeiler and Leman go Machine Learning: The Story so farcited by 121
  21. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  22. 2020Predicting Neural Network Accuracy from Weightscited by 107
  23. 2020Expressive Power of Invariant and Equivariant Graph Neural Networkscited by 100
  24. 2020On Learning Sets of Symmetric Elementscited by 97
  25. 2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 90
  26. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  27. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  28. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  29. 2018Deep Models of Interactions Across Setscited by 68
  30. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  31. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  32. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
  33. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  34. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  35. 2022Signal Processing for Implicit Neural Representationscited by 46
  36. 2019Neural Network Branching for Neural Network Verificationcited by 45
  37. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  38. 2015Understanding symmetries in deep networkscited by 17

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 26; 2025: 34; 2026: 31.

106 papers.

Sort:

By area:

  1. 2023Permutation Equivariant Neural Functionalscited by 69
  2. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  3. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  4. 2023Neural Functional Transformerscited by 50
  5. 2023Mosaic-SDF for 3D Generative Modelscited by 48
  6. 2024Neural Network Diffusioncited by 39
  7. 2023Equivariant Deep Weight Space Alignmentcited by 34
  8. 2025Symmetry in Neural Network Parameter Spacescited by 31
  9. 2024Universal Neural Functionalscited by 25
  10. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  11. 2024Improved Generalization of Weight Space Networks via Augmentationscited by 24
  12. 2024Scale Equivariant Graph Metanetworkscited by 23
38 more
  1. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 22
  2. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  3. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  4. 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
  5. 2025Update Your Transformer to the Latest Release: Re-Basin of Task Vectorscited by 17
  6. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  7. 2025Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViTcited by 16
  8. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
  9. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 13
  10. 2023How to Train Neural Field Representations: A Comprehensive Study and Benchmarkcited by 12
  11. 2023Robot Fleet Learning via Policy Mergingcited by 11
  12. 2024Unsupervised Model Tree Heritage Recoverycited by 11
  13. 2023Set-based Neural Network Encoding Without Weight Tyingcited by 9
  14. 2023Neural Processing of Tri-Plane Hybrid Neural Fieldscited by 8
  15. 2024Symmetry Discovery for Different Data Typescited by 7
  16. 2024Learning on Model Weights using Tree Expertscited by 7
  17. 2025End-to-End Implicit Neural Representations for Classificationcited by 7
  18. 2025GradMetaNet: An Equivariant Architecture for Learning on Gradientscited by 7
  19. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 7
  20. 2024Connecting NeRFs, Images, and Textcited by 6
  21. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  22. 2024Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networkscited by 6
  23. 2025MedFuncta: A Unified Framework for Learning Efficient Medical Neural Fieldscited by 6
  24. 2025Model Unmerging: Making Your Models Unmergeable for Secure Model Sharingcited by 6
  25. 2023Deep Learning on Object-centric 3D Neural Fieldscited by 5
  26. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  27. 2024Towards Croppable Implicit Neural Representationscited by 5
  28. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
  29. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  30. 2025Text2Weight: Bridging Natural Language and Neural Network Weight Spacescited by 5
  31. 2023Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Modelscited by 4
  32. 2023Exploring the Complexity of Deep Neural Networks through Functional Equivalencecited by 4
  33. 2023Representation-Driven Reinforcement Learningcited by 4
  34. 2025Learning Model Representations Using Publicly Available Model Hubscited by 4
  35. 2026W2T: LoRA Weights Already Know What They Can Docited by 4
  36. 2024Vanishing Feature: Diagnosing Model Merging and Beyondcited by 3
  37. 2024Encoding Semantic Priors into the Weights of Implicit Neural Representationcited by 3
  38. 2025ARC: Anchored Representation Clouds for High-Resolution INR Classificationcited by 3

50 most cited of 106 All 106 (JSON)