mlsanityData to 14 Oct 2026

Constraining the outputs of ReLU neural networks

arXiv 2508.03867 ↗First posted 5 Aug 2025math.AG cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce a class of algebraic varieties naturally associated with ReLU neural networks, arising from the piecewise linear structure of their outputs across activation regions in input space, and the piecewise multilinear structure in parameter space. By analyzing the rank constraints on the network outputs within each activation region, we derive polynomial equations that characterize the functions representable by the network. We further investigate conditions under which these varieties attain their expected dimension, providing insight into the expressive and structural properties of ReLU networks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

36 of 45 references.

References · 36

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  3. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  4. 2016Benefits of depth in neural networkscited by 250
  5. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  6. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  7. 2018Tropical Geometry of Deep Neural Networkscited by 95
  8. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  9. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  10. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  11. 2019Reverse-Engineering Deep ReLU Networkscited by 83
  12. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 56
24 more
  1. 2024Deep Networks Always Grok and Here is Whycited by 51
  2. 2019Minimum Description Length Revisitedcited by 37
  3. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  4. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  5. 2023Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopescited by 36
  6. 2021Geometry of Linear Convolutional Networkscited by 27
  7. 2018A Tropical Approach to Neural Networks with Piecewise Linear Activationscited by 26
  8. 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
  9. 2024The Real Tropical Geometry of Neural Networkscited by 24
  10. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  11. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  12. 2020Affine symmetries and neural network identifiabilitycited by 18
  13. 2023Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscapecited by 17
  14. 2020Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networkscited by 15
  15. 2021On the Expected Complexity of Maxout Networkscited by 15
  16. 2018Restricted Boltzmann Machines: Introduction and Reviewcited by 14
  17. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  18. 2022Local Identifiability of Deep ReLU Neural Networks: the Theorycited by 12
  19. 2017The Multilinear Structure of ReLU Networkscited by 9
  20. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  21. 2024Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regressioncited by 6
  22. 2025Approximating Latent Manifolds in Neural Networks via Vanishing Idealscited by 5
  23. 2019Best k-layer neural network approximationscited by 4
  24. 2022A Combinatorial Perspective on the Optimization of Shallow ReLU Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Most ReLU Networks Admit Identifiable Parameterscited by 6
  2. 2026Robustness Verification of Polynomial Neural Networkscited by 3
  3. 2026Algebraic Invariants of Lightning Self-Attentioncited by 1