# Scale Equivariant Graph Metanetworks

arXiv: https://arxiv.org/abs/2406.10685 · HTML: https://mlsanity.com/paper/2406.10685/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 85 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 30 of 65)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Principal Neighbourhood Aggregation for Graph Nets](https://mlsanity.com/paper/2004.05718/) (2020)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [Sign and Basis Invariant Networks for Spectral Graph Representation Learning](https://mlsanity.com/paper/2202.13013/) (2022)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [Frame Averaging for Invariant and Equivariant Network Design](https://mlsanity.com/paper/2110.03336/) (2021)
- [Universal approximations of invariant maps by neural networks](https://mlsanity.com/paper/1804.10306/) (2018)
- [Path-SGD: Path-Normalized Optimization in Deep Neural Networks](https://mlsanity.com/paper/1506.02617/) (2015)
- [Predicting Neural Network Accuracy from Weights](https://mlsanity.com/paper/2002.11448/) (2020)
- [Equivariance with Learned Canonicalization Functions](https://mlsanity.com/paper/2211.06489/) (2022)

## Built on by (most cited first, 24 of 24)

- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models](https://mlsanity.com/paper/2410.04207/) (2024)
- [Monomial Matrix Group Equivariant Neural Functional Networks](https://mlsanity.com/paper/2409.11697/) (2024)
- [Deep Linear Probe Generators for Weight Space Learning](https://mlsanity.com/paper/2410.10811/) (2024)
- [Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion](https://mlsanity.com/paper/2502.00264/) (2025)
- [Generative Modeling of Weights: Generalization or Memorization?](https://mlsanity.com/paper/2506.07998/) (2025)
- [End-to-End Implicit Neural Representations for Classification](https://mlsanity.com/paper/2503.18123/) (2025)
- [GradMetaNet: An Equivariant Architecture for Learning on Gradients](https://mlsanity.com/paper/2507.01649/) (2025)
- [ARC: Anchored Representation Clouds for High-Resolution INR Classification](https://mlsanity.com/paper/2503.15156/) (2025)
- [Improving Learning to Optimize Using Parameter Symmetries](https://mlsanity.com/paper/2504.15399/) (2025)
- [Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruning](https://mlsanity.com/paper/2506.12041/) (2025)
- [A Graph Meta-Network for Learning on Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2602.16316/) (2026)
- [Quasi-Equivariant Metanetworks](https://mlsanity.com/paper/2604.23720/) (2026)
- [On the Internal Representations of Graph Metanetworks](https://mlsanity.com/paper/2503.09120/) (2025)
- [DUN-SRE: Deep Unrolling Network with Spatiotemporal Rotation Equivariance for Dynamic MRI Reconstruction](https://mlsanity.com/paper/2506.10309/) (2025)
- [Graph Lineages and Skeletal Graph Products](https://mlsanity.com/paper/2508.00197/) (2025)
- [Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance](https://mlsanity.com/paper/2512.15469/) (2025)
- [Is Class Signal Clustered or Routed in Task-Induced Implicit Neural Representation Weight Spaces?](https://mlsanity.com/paper/2605.08281/) (2026)
- [Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification](https://mlsanity.com/paper/2607.29463/) (2026)
- [Permutation-Equivariant Flow Matching for Alignment-Free Neural Weight Generation](https://mlsanity.com/paper/2609.32833/) (2026)
- [Discovering Symmetries in Neural Network Parameter Spaces](https://mlsanity.com/paper/2609.33527/) (2026)
- [Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning](https://mlsanity.com/paper/2609.33901/) (2026)
- [CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations](https://mlsanity.com/paper/2610.01649/) (2026)
- [Graph Metanetworks for Transformer Based Language Models](https://mlsanity.com/paper/2610.18471/) (2026)

## Abstract (arXiv)

This paper pertains to an emerging machine learning paradigm: learning higher-order functions, i.e. functions whose inputs are functions themselves, textitparticularly when these inputs are Neural Networks (NNs). With the growing interest in architectures that process NNs, a recurring design principle has permeated the field: adhering to the permutation symmetries arising from the connectionist structure of NNs. textitHowever, are these the sole symmetries present in NN parameterizations? Zooming into most practical activation functions (e.g. sine, ReLU, tanh) answers this question negatively and gives rise to intriguing new symmetries, which we collectively refer to as textitscaling symmetries, that is, non-zero scalar multiplications and divisions of weights and biases. In this work, we propose textitScale Equivariant Graph MetaNetworks - ScaleGMNs, a framework that adapts the Graph Metanetwork (message-passing) paradigm by incorporating scaling symmetries and thus rendering neuron and edge representations equivariant to valid scalings. We introduce novel building blocks, of independent technical interest, that allow for equivariance or invariance with respect to individual scalar multipliers or their product and use them in all components of ScaleGMN. Furthermore, we prove that, under certain expressivity conditions, ScaleGMN can simulate the forward and backward pass of any input feedforward neural network. Experimental results demonstrate that our method advances the state-of-the-art performance for several datasets and activation functions, highlighting the power of scaling symmetries as an inductive bias for NN processing. The source code is publicly available at https://github.com/jkalogero/scalegmn.