# Graph Metanetworks for Processing Diverse Neural Architectures

arXiv: https://arxiv.org/abs/2312.04501 · HTML: https://mlsanity.com/paper/2312.04501/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 64 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [DiGress: Discrete Denoising diffusion for graph generation](https://mlsanity.com/paper/2209.14734/) (2022)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Pure Transformers are Powerful Graph Learners](https://mlsanity.com/paper/2207.02505/) (2022)
- [A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups](https://mlsanity.com/paper/2104.09459/) (2021)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [Sign and Basis Invariant Networks for Spectral Graph Representation Learning](https://mlsanity.com/paper/2202.13013/) (2022)
- [Graph Inductive Biases in Transformers without Message Passing](https://mlsanity.com/paper/2305.17589/) (2023)
- [HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion](https://mlsanity.com/paper/2303.17015/) (2023)
- [Equivariance Through Parameter-Sharing](https://mlsanity.com/paper/1702.08389/) (2017)
- [Predicting Neural Network Accuracy from Weights](https://mlsanity.com/paper/2002.11448/) (2020)

## Built on by (most cited first, 30 of 57)

- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Universal Neural Functionals](https://mlsanity.com/paper/2402.05232/) (2024)
- [We Should Chart an Atlas of All the World's Models](https://mlsanity.com/paper/2503.10633/) (2025)
- [SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass](https://mlsanity.com/paper/2602.06358/) (2026)
- [Improved Generalization of Weight Space Networks via Augmentations](https://mlsanity.com/paper/2402.04081/) (2024)
- [Scale Equivariant Graph Metanetworks](https://mlsanity.com/paper/2406.10685/) (2024)
- [Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models](https://mlsanity.com/paper/2410.04207/) (2024)
- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [Grounding Continuous Representations in Geometry: Equivariant Neural Fields](https://mlsanity.com/paper/2406.05753/) (2024)
- [Monomial Matrix Group Equivariant Neural Functional Networks](https://mlsanity.com/paper/2409.11697/) (2024)
- [Deep Linear Probe Generators for Weight Space Learning](https://mlsanity.com/paper/2410.10811/) (2024)
- [Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion](https://mlsanity.com/paper/2502.00264/) (2025)
- [Generative Modeling of Weights: Generalization or Memorization?](https://mlsanity.com/paper/2506.07998/) (2025)
- [Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning](https://mlsanity.com/paper/2506.10973/) (2025)
- [Unsupervised Model Tree Heritage Recovery](https://mlsanity.com/paper/2405.18432/) (2024)
- [Learning on Model Weights using Tree Experts](https://mlsanity.com/paper/2410.13569/) (2024)
- [End-to-End Implicit Neural Representations for Classification](https://mlsanity.com/paper/2503.18123/) (2025)
- [GradMetaNet: An Equivariant Architecture for Learning on Gradients](https://mlsanity.com/paper/2507.01649/) (2025)
- [Connecting NeRFs, Images, and Text](https://mlsanity.com/paper/2404.07993/) (2024)
- [LLaNA: Large Language and NeRF Assistant](https://mlsanity.com/paper/2406.11840/) (2024)
- [Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks](https://mlsanity.com/paper/2408.05496/) (2024)
- [Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction](https://mlsanity.com/paper/2503.17138/) (2025)
- [Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond](https://mlsanity.com/paper/2505.04621/) (2025)
- [Position: Weight Space Should Be a First-Class Generative AI Modality](https://mlsanity.com/paper/2605.18632/) (2026)
- [Neural Spacetimes for DAG Representation Learning](https://mlsanity.com/paper/2408.13885/) (2024)
- [What makes a good feedforward computational graph?](https://mlsanity.com/paper/2502.06751/) (2025)
- [Text2Weight: Bridging Natural Language and Neural Network Weight Spaces](https://mlsanity.com/paper/2508.13633/) (2025)
- [Learning Model Representations Using Publicly Available Model Hubs](https://mlsanity.com/paper/2510.02096/) (2025)
- [Tensor learning with orthogonal, Lorentz, and symplectic symmetries](https://mlsanity.com/paper/2406.01552/) (2024)
- [A Model Zoo of Vision Transformers](https://mlsanity.com/paper/2504.10231/) (2025)

## Abstract (arXiv)

Neural networks efficiently encode learned information within their parameters. Consequently, many tasks can be unified by treating neural networks themselves as input data. When doing so, recent studies demonstrated the importance of accounting for the symmetries and geometry of parameter spaces. However, those works developed architectures tailored to specific networks such as MLPs and CNNs without normalization layers, and generalizing such architectures to other types of networks can be challenging. In this work, we overcome these challenges by building new metanetworks - neural networks that take weights from other neural networks as input. Put simply, we carefully build graphs representing the input neural networks and process the graphs using graph neural networks. Our approach, Graph Metanetworks (GMNs), generalizes to neural architectures where competing methods struggle, such as multi-head attention layers, normalization layers, convolutional layers, ResNet blocks, and group-equivariant linear layers. We prove that GMNs are expressive and equivariant to parameter permutation symmetries that leave the input neural network functions unchanged. We validate the effectiveness of our method on several metanetwork tasks over diverse neural network architectures.