# Identifiable Equivariant Networks are Layerwise Equivariant

arXiv: https://arxiv.org/abs/2601.21645 · HTML: https://mlsanity.com/paper/2601.21645/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 30 of 34)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [On the Universality of Invariant Networks](https://mlsanity.com/paper/1901.09342/) (2019)
- [Universal approximations of invariant maps by neural networks](https://mlsanity.com/paper/1804.10306/) (2018)
- [On the Universality of Rotation Equivariant Point Cloud Networks](https://mlsanity.com/paper/2010.02449/) (2020)
- [On the Expressive Power of Deep Polynomial Neural Networks](https://mlsanity.com/paper/1905.12207/) (2019)
- [On the Symmetries of Deep Learning Models and their Internal Representations](https://mlsanity.com/paper/2205.14258/) (2022)
- [The Lie Derivative for Measuring Learned Equivariance](https://mlsanity.com/paper/2210.02984/) (2022)
- [Does equivariance matter at scale?](https://mlsanity.com/paper/2410.23179/) (2024)
- [Universal Equivariant Multilayer Perceptrons](https://mlsanity.com/paper/2002.02912/) (2020)
- [Backprop as Functor: A compositional perspective on supervised learning](https://mlsanity.com/paper/1711.10455/) (2017)
- [Hidden symmetries of ReLU networks](https://mlsanity.com/paper/2306.06179/) (2023)
- [Equivalence of approximation by convolutional neural networks and fully-connected networks](https://mlsanity.com/paper/1809.00973/) (2018)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Pure and Spurious Critical Points: a Geometric Study of Linear Networks](https://mlsanity.com/paper/1910.01671/) (2019)
- [Position: Categorical Deep Learning is an Algebraic Theory of All Architectures](https://mlsanity.com/paper/2402.15332/) (2024)
- [Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks](https://mlsanity.com/paper/2312.08550/) (2023)
- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometry](https://mlsanity.com/paper/2501.18915/) (2025)
- [Affine symmetries and neural network identifiability](https://mlsanity.com/paper/2006.11727/) (2020)
- [Optimization Dynamics of Equivariant and Augmented Neural Networks](https://mlsanity.com/paper/2303.13458/) (2023)
- [Geometry of Lightning Self-Attention: Identifiability and Dimension](https://mlsanity.com/paper/2408.17221/) (2024)
- [Neural network identifiability for a family of sigmoidal nonlinearities](https://mlsanity.com/paper/1906.06994/) (2019)
- [Activation degree thresholds and expressiveness of polynomial neural networks](https://mlsanity.com/paper/2408.04569/) (2024)
- [Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groups](https://mlsanity.com/paper/2205.14819/) (2022)
- [On the Geometry and Optimization of Polynomial Convolutional Networks](https://mlsanity.com/paper/2410.00722/) (2024)
- [Investigating how ReLU-networks encode symmetries](https://mlsanity.com/paper/2305.17017/) (2023)
- [A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models](https://mlsanity.com/paper/2506.02269/) (2025)

## Built on by (most cited first, 7 of 7)

- [Most ReLU Networks Admit Identifiable Parameters](https://mlsanity.com/paper/2605.03601/) (2026)
- [The Geometry of Polynomial Group Convolutional Neural Networks](https://mlsanity.com/paper/2603.29566/) (2026)
- [Conservation Laws from Data Symmetry in Neural Networks](https://mlsanity.com/paper/2606.10913/) (2026)
- [Singular Learning and Occam's Razor in Deep Monomial Networks](https://mlsanity.com/paper/2606.28464/) (2026)
- [Boosting Data Augmentation with Stochastic Weight Averaging](https://mlsanity.com/paper/2608.14373/) (2026)
- [Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks](https://mlsanity.com/paper/2608.27113/) (2026)
- [Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention](https://mlsanity.com/paper/2608.30417/) (2026)

## Abstract (arXiv)

We investigate the relation between end-to-end equivariance and layerwise equivariance in deep neural networks. We prove the following: For a network whose end-to-end function is equivariant with respect to group actions on the input and output spaces, there is a parameter choice yielding the same end-to-end function such that its layers are equivariant with respect to some group actions on the latent spaces. Our result assumes that the parameters of the model are identifiable in an appropriate sense. This identifiability property has been established in the literature for a large class of networks, to which our results apply immediately, while it is conjectural for others. The theory we develop is grounded in an abstract formalism, and is therefore architecture-agnostic. Overall, our results provide a mathematical explanation for the emergence of equivariant structures in the weights of neural networks during training -- a phenomenon that is consistently observed in practice.