# Equivariant Polynomial Functional Networks

arXiv: https://arxiv.org/abs/2410.04213 · HTML: https://mlsanity.com/paper/2410.04213/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2024.

## Built on by (most cited first, 11 of 11)

- [On Linear Mode Connectivity of Mixture-of-Experts Architectures](https://mlsanity.com/paper/2509.11348/) (2025)
- [MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling](https://mlsanity.com/paper/2503.11144/) (2025)
- [Distance-Based Tree-Sliced Wasserstein Distance](https://mlsanity.com/paper/2503.11050/) (2025)
- [Tree-Sliced Wasserstein Distance with Nonlinear Projection](https://mlsanity.com/paper/2505.00968/) (2025)
- [Spherical Tree-Sliced Wasserstein Distance](https://mlsanity.com/paper/2503.11249/) (2025)
- [Improving Learning to Optimize Using Parameter Symmetries](https://mlsanity.com/paper/2504.15399/) (2025)
- [Quasi-Equivariant Metanetworks](https://mlsanity.com/paper/2604.23720/) (2026)
- [Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance](https://mlsanity.com/paper/2512.15469/) (2025)
- [Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning](https://mlsanity.com/paper/2609.33901/) (2026)
- [CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations](https://mlsanity.com/paper/2610.01649/) (2026)
- [Graph Metanetworks for Transformer Based Language Models](https://mlsanity.com/paper/2610.18471/) (2026)

## Abstract (arXiv)

Neural Functional Networks (NFNs) have gained increasing interest due to their wide range of applications, including extracting information from implicit representations of data, editing network weights, and evaluating policies. A key design principle of NFNs is their adherence to the permutation and scaling symmetries inherent in the connectionist structure of the input neural networks. Recent NFNs have been proposed with permutation and scaling equivariance based on either graph-based message-passing mechanisms or parameter-sharing mechanisms. However, graph-based equivariant NFNs suffer from high memory consumption and long running times. On the other hand, parameter-sharing-based NFNs built upon equivariant linear layers exhibit lower memory consumption and faster running time, yet their expressivity is limited due to the large size of the symmetric group of the input neural networks. The challenge of designing a permutation and scaling equivariant NFN that maintains low memory consumption and running time while preserving expressivity remains unresolved. In this paper, we propose a novel solution with the development of MAGEP-NFN (Monomial mAtrix Group Equivariant Polynomial NFN). Our approach follows the parameter-sharing mechanism but differs from previous works by constructing a nonlinear equivariant layer represented as a polynomial in the input weights. This polynomial formulation enables us to incorporate additional relationships between weights from different input hidden layers, enhancing the model's expressivity while keeping memory consumption and running time low, thereby addressing the aforementioned challenge. We provide empirical evidence demonstrating that MAGEP-NFN achieves competitive performance and efficiency compared to existing baselines.