mlsanityData to 14 Oct 2026

Hidden symmetries of ReLU networks

arXiv 2306.06179 ↗First posted 9 Jun 2023cs.LG math.CO math.GTcited by 39

Not on the map. Why?

Abstract (arXiv)

The parameter space for any fixed architecture of feedforward ReLU neural networks serves as a proxy during training for the associated class of functions - but how faithful is this representation? It is known that many different parameter settings can determine the same function. Moreover, the degree of this redundancy is inhomogeneous: for some networks, the only symmetries are permutation of neurons in a layer and positive scaling of parameters at a neuron, while other networks admit additional hidden symmetries. In this work, we prove that, for any network architecture where no layer is narrower than the input, there exist parameter settings with no hidden symmetries. We also describe a number of mechanisms through which hidden symmetries can arise, and empirically approximate the functional dimension of different network architectures at initialization. These experiments indicate that the probability that a network has no hidden symmetries decreases towards 0 as depth increases, while increasing towards 1 as width and input dimension increase.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

25 of 36 references.

References · 25

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  3. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  4. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  5. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  6. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  7. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  8. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  9. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  10. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  11. 2018Model Reconstruction from Model Explanationscited by 86
  12. 2019Reverse-Engineering Deep ReLU Networkscited by 83
13 more
  1. 2020Cryptanalytic Extraction of Neural Network Modelscited by 68
  2. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  3. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 63
  4. 2020On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networkscited by 35
  5. 2018G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Spacecited by 27
  6. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25
  7. 2022Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networkscited by 22
  8. 2022Functional dimension of feedforward ReLU neural networkscited by 21
  9. 2020The Representation Theory of Neural Networkscited by 19
  10. 2020Neural Teleportationcited by 18
  11. 2015Symmetry-invariant optimization in deep networkscited by 15
  12. 2022Local and global topological complexity measures OF ReLU neural network functionscited by 8
  13. 2022Quiver neural networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 9; 2026: 18.

39 papers.

Sort:

By area:

  1. 2025Symmetry in Neural Network Parameter Spacescited by 36
  2. 2024Training-Free Pretrained Model Mergingcited by 32
  3. 2024The Real Tropical Geometry of Neural Networkscited by 24
  4. 2024Scale Equivariant Graph Metanetworkscited by 24
  5. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  6. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 17
  7. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  8. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  9. 2025Token embeddings violate the manifold hypothesiscited by 16
  10. 2023A Symmetry-Aware Exploration of Bayesian Neural Network Posteriorscited by 12
  11. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  12. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
27 more
  1. 2023Geometric structure of Deep Learning networks and construction of global mathcal L2 minimizerscited by 6
  2. 2024Where Do Large Learning Rates Lead Us?cited by 6
  3. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  4. 2026Most ReLU Networks Admit Identifiable Parameterscited by 6
  5. 2024Interpretable global minima of deep ReLU neural networks on sequentially separable datacited by 4
  6. 2024Relative Representations: Topological and Geometric Perspectivescited by 4
  7. 2025On Space Folds of ReLU Neural Networkscited by 3
  8. 2026A Complete Symmetry Classification of Shallow ReLU Networkscited by 3
  9. 2026The Symmetries of Three-Layer ReLU Networkscited by 3
  10. 2025Relating Piecewise Linear Kolmogorov Arnold Networks to ReLU Networkscited by 2
  11. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  12. 2026On the Epistemic Uncertainty of Overparametrized Neural Networkscited by 2
  13. 2024On Functional Dimension and Persistent Pseudodimensioncited by 1
  14. 2026Complete Identification of Deep ReLU Networks through Lukasiewicz Logiccited by 1
  15. 2026Quasi-Equivariant Metanetworkscited by 1
  16. 2026Conservation Laws from Data Symmetry in Neural Networkscited by 1
  17. 2024Hyper-Representations: Learning from Populations of Neural Networkscited by 0
  18. 2025Data Augmentation Techniques to Reverse-Engineer Neural Network Weights from Input-Output Queriescited by 0
  19. 2026The Role of Symmetry in Optimizing Overparameterized Networkscited by 0
  20. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  21. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  22. 2026Weight-Space Mixture-of-Experts for Implicit Neural Representation Classificationcited by 0
  23. 2026Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networkscited by 0
  24. 2026Functional Degeneracy in Neural Networks: Measurement and Pruningcited by 0
  25. 2026Symmetry-quotient Flatness and Generalizationcited by 0
  26. 2026Discovering Symmetries in Neural Network Parameter Spacescited by 0
  27. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0