mlsanityData to 14 Oct 2026

Most ReLU Networks Admit Identifiable Parameters

arXiv 2605.03601 ↗First posted 5 May 2026cs.LG cs.DM math.COcited by 6

Not on the map. Why?

Abstract (arXiv)

We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation. To analyze hidden redundancies beyond these standard symmetries, we introduce a framework based on weighted polyhedral complexes. Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters. This implies that the functional dimension of every such architecture is exactly the number of parameters minus the number of hidden neurons. We further show that minimal functional representations can still have non-trivial parameter redundancies. Finally, we establish a generic depth hierarchy, whereby for an open set of parameters the realized function cannot be represented generically by any shallower network.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 46 of its 62 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

46 of 62 references.

References · 46

  1. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  2. 2016On the Expressive Power of Deep Neural Networkscited by 306
  3. 2016Benefits of depth in neural networkscited by 250
  4. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  5. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  6. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  7. 2018Tropical Geometry of Deep Neural Networkscited by 95
  8. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  9. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  10. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  11. 2019Reverse-Engineering Deep ReLU Networkscited by 83
  12. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 56
34 more
  1. 2019The Geometry of Deep Networks: Power Diagram Subdivisioncited by 53
  2. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  3. 2023Hidden symmetries of ReLU networkscited by 39
  4. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  5. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  6. 2023Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopescited by 36
  7. 2025Symmetry in Neural Network Parameter Spacescited by 36
  8. 2020On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networkscited by 35
  9. 2018A Tropical Approach to Neural Networks with Piecewise Linear Activationscited by 26
  10. 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
  11. 2024The Real Tropical Geometry of Neural Networkscited by 24
  12. 2022Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networkscited by 22
  13. 2024Equivariant Neural Functional Networks for Transformerscited by 22
  14. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  15. 2020Affine symmetries and neural network identifiabilitycited by 18
  16. 2025On the Expressiveness of Rational ReLU Neural Networks With Bounded Depthcited by 18
  17. 2023Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscapecited by 17
  18. 2018Restricted Boltzmann Machines: Introduction and Reviewcited by 14
  19. 2024Activation degree thresholds and expressiveness of polynomial neural networkscited by 14
  20. 2021On minimal representations of shallow ReLU networkscited by 13
  21. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  22. 2025Better Neural Network Expressivity: Subdividing the Simplexcited by 12
  23. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  24. 2025Depth-Bounds for Neural Networks via the Braid Arrangementcited by 11
  25. 2023Topological Expressivity of ReLU Neural Networkscited by 10
  26. 2024Decomposition Polyhedra of Piecewise Linear Functionscited by 9
  27. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  28. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
  29. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  30. 2023On the existence of optimal shallow feedforward networks with ReLU activationcited by 5
  31. 2015Dimension of Marginals of Kronecker Product Modelscited by 4
  32. 2025Constraining the outputs of ReLU neural networkscited by 3
  33. 2026A Depth Hierarchy for Computing the Maximum in ReLU Networks via Extremal Graph Theorycited by 3
  34. 2026A Complete Symmetry Classification of Shallow ReLU Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026The Symmetries of Three-Layer ReLU Networkscited by 3
  2. 2026Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapsecited by 2
  3. 2026Conservation Laws from Data Symmetry in Neural Networkscited by 1
  4. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  5. 2026Every Layer Counts: An Exponential L2 Depth Hierarchy for ReLU Networkscited by 0
  6. 2026Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networkscited by 0