mlsanityData to 7 Oct 2026

The Geometric Structure of Fully-Connected ReLU Layers

arXiv 2310.03482 ↗First posted 5 Oct 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We formalize and interpret the geometric structure of d-dimensional fully connected ReLU layers in neural networks. The parameters of a ReLU layer induce a natural partition of the input domain, such that the ReLU layer can be significantly simplified in each sector of the partition. This leads to a geometric interpretation of a ReLU layer as a projection onto a polyhedral cone followed by an affine transformation, in line with the description in [doi:10.48550/arXiv.1905.08922] for convolutional networks with ReLU activations. Further, this structure facilitates simplified expressions for preimages of the intersection between partition sectors and hyperplanes, which is useful when describing decision boundaries in a classification setting. We investigate this in detail for a feed-forward network with one hidden ReLU-layer, where we provide results on the geometric complexity of the decision boundary generated by such networks, as well as proving that modulo an affine transformation, such a network can only generate d different decision boundaries. Finally, the effect of adding more layers to the network is discussed.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 11 of its 18 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

11 of 18 references.

References · 11

  1. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  2. 2013Maxout Networkscited by 186
  3. 2018Robustness via curvature regularization, and vice versacited by 174
  4. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  5. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  6. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 120
  7. 2020Topology of deep neural networkscited by 111
  8. 2018Tropical Geometry of Deep Neural Networkscited by 92
  9. 2020On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspectivecited by 25
  10. 2016Efficient Representation of Low-Dimensional Manifolds using Deep Networkscited by 13
  11. 2019Geometry of Deep Convolutional Networkscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2025Marching Neurons: Accurate Surface Extraction for Neural Implicit Shapescited by 3
  2. 2024Defining Neural Network Architecture through Polytope Structures of Datasetcited by 2
  3. 2024Implicit Hypersurface Approximation Capacity in Deep ReLU Networkscited by 0
  4. 2025Explicit neural network classifiers for non-separable datacited by 0