# Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups

arXiv: https://arxiv.org/abs/2309.13736 · HTML: https://mlsanity.com/paper/2309.13736/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 24 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [B-Spline CNNs on Lie Groups](https://mlsanity.com/paper/1909.12057/) (2019)
- [Roto-Translation Covariant Convolutional Networks for Medical Image Analysis](https://mlsanity.com/paper/1804.03393/) (2018)
- [Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds](https://mlsanity.com/paper/2106.06020/) (2021)
- [Pure and Spurious Critical Points: a Geometric Study of Linear Networks](https://mlsanity.com/paper/1910.01671/) (2019)
- [Geometry of Linear Convolutional Networks](https://mlsanity.com/paper/2108.01538/) (2021)
- [Convergence of gradient descent for learning linear neural networks](https://mlsanity.com/paper/2108.02040/) (2021)
- [What is an equivariant neural network?](https://mlsanity.com/paper/2205.07362/) (2022)
- [Multiresolution Equivariant Graph Variational Autoencoder](https://mlsanity.com/paper/2106.00967/) (2021)
- [Function Space and Critical Points of Linear Convolutional Networks](https://mlsanity.com/paper/2304.05752/) (2023)
- [The Euclidean distance degree of an algebraic variety](https://mlsanity.com/paper/1309.0049/) (2013)

## Built on by (most cited first, 5 of 5)

- [Training Dynamics of In-Context Learning in Linear Attention](https://mlsanity.com/paper/2501.16265/) (2025)
- [Activation degree thresholds and expressiveness of polynomial neural networks](https://mlsanity.com/paper/2408.04569/) (2024)
- [Algebraic Complexity and Neurovariety of Linear Convolutional Networks](https://mlsanity.com/paper/2401.16613/) (2024)
- [Understanding Learning Invariance in Deep Linear Networks](https://mlsanity.com/paper/2506.13714/) (2025)
- [Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention](https://mlsanity.com/paper/2608.30417/) (2026)

## Abstract (arXiv)

The set of functions parameterized by a linear fully-connected neural network is a determinantal variety. We investigate the subvariety of functions that are equivariant or invariant under the action of a permutation group. Examples of such group actions are translations or 90^circ rotations on images. We describe such equivariant or invariant subvarieties as direct products of determinantal varieties, from which we deduce their dimension, degree, Euclidean distance degree, and their singularities. We fully characterize invariance for arbitrary permutation groups, and equivariance for cyclic groups. We draw conclusions for the parameterization and the design of equivariant and invariant linear networks in terms of sparsity and weight-sharing properties. We prove that all invariant linear functions can be parameterized by a single linear autoencoder with a weight-sharing property imposed by the cycle decomposition of the considered permutation. The space of rank-bounded equivariant functions has several irreducible components, so it can not be parameterized by a single network-but each irreducible component can. Finally, we show that minimizing the squared-error loss on our invariant or equivariant networks reduces to minimizing the Euclidean distance from determinantal varieties via the Eckart-Young theorem.