The Symmetries of Three-Layer ReLU Networks
Not on the map. Why?
Abstract (arXiv)
We develop a framework for analyzing parameter symmetries in deep ReLU networks and obtain a complete characterization of the generic parameter fibers for three-layer bottleneck architectures. Our approach provides explicit semi-algebraic descriptions of these fibers and yields a polynomial time algorithm for deciding functional equivalence of two parameters. The symmetries include discrete and continuous transformations arising from layer composition, and depend on whether deeper layers hide or preserve geometric structure from preceding layers. Finally, we show that some of these symmetries induce local conservation laws along gradient flow, while others do not.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 18 May 2026.
Builds on
25 of 38 references.
References · 25
- 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
- 2019Reverse-Engineering Deep ReLU Networkscited by 81
- 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
- 2023Hidden symmetries of ReLU networkscited by 38
- 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
- 2020On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networkscited by 34
- 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
- 2025Symmetry in Neural Network Parameter Spacescited by 31
- 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
- 2022Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networkscited by 21
- 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
- 2022Functional dimension of feedforward ReLU neural networkscited by 20
13 more
- 2021Noether: The More Things Change, the More Stay the Samecited by 19
- 2020Affine symmetries and neural network identifiabilitycited by 17
- 2019How degenerate is the parametrization of neural networks with the ReLU activation function?cited by 15
- 2024Activation degree thresholds and expressiveness of polynomial neural networkscited by 14
- 2021On minimal representations of shallow ReLU networkscited by 13
- 2022Local Identifiability of Deep ReLU Neural Networks: the Theorycited by 12
- 2025Identifiability of Deep Polynomial Neural Networkscited by 12
- 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
- 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
- 2023On the existence of optimal shallow feedforward networks with ReLU activationcited by 5
- 2026Most ReLU Networks Admit Identifiable Parameterscited by 5
- 2025Parameterized Hardness of Zonotope Containment and Neural Network Verificationcited by 4
- 2026A Complete Symmetry Classification of Shallow ReLU Networkscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.