# The Geometric Structure of Fully-Connected ReLU Layers

arXiv: https://arxiv.org/abs/2310.03482 · HTML: https://mlsanity.com/paper/2310.03482/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 11 of 11)

- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Robustness via curvature regularization, and vice versa](https://mlsanity.com/paper/1811.09716/) (2018)
- [Deep ReLU Networks Have Surprisingly Few Activation Patterns](https://mlsanity.com/paper/1906.00904/) (2019)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Approximating Continuous Functions by ReLU Nets of Minimal Width](https://mlsanity.com/paper/1710.11278/) (2017)
- [Topology of deep neural networks](https://mlsanity.com/paper/2004.06093/) (2020)
- [Tropical Geometry of Deep Neural Networks](https://mlsanity.com/paper/1805.07091/) (2018)
- [On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspective](https://mlsanity.com/paper/2002.08838/) (2020)
- [Efficient Representation of Low-Dimensional Manifolds using Deep Networks](https://mlsanity.com/paper/1602.04723/) (2016)
- [Geometry of Deep Convolutional Networks](https://mlsanity.com/paper/1905.08922/) (2019)

## Built on by (most cited first, 4 of 4)

- [Marching Neurons: Accurate Surface Extraction for Neural Implicit Shapes](https://mlsanity.com/paper/2509.21007/) (2025)
- [Defining Neural Network Architecture through Polytope Structures of Dataset](https://mlsanity.com/paper/2402.02407/) (2024)
- [Implicit Hypersurface Approximation Capacity in Deep ReLU Networks](https://mlsanity.com/paper/2407.03851/) (2024)
- [Explicit neural network classifiers for non-separable data](https://mlsanity.com/paper/2504.18710/) (2025)

## Abstract (arXiv)

We formalize and interpret the geometric structure of d-dimensional fully connected ReLU layers in neural networks. The parameters of a ReLU layer induce a natural partition of the input domain, such that the ReLU layer can be significantly simplified in each sector of the partition. This leads to a geometric interpretation of a ReLU layer as a projection onto a polyhedral cone followed by an affine transformation, in line with the description in [doi:10.48550/arXiv.1905.08922] for convolutional networks with ReLU activations. Further, this structure facilitates simplified expressions for preimages of the intersection between partition sectors and hyperplanes, which is useful when describing decision boundaries in a classification setting. We investigate this in detail for a feed-forward network with one hidden ReLU-layer, where we provide results on the geometric complexity of the decision boundary generated by such networks, as well as proving that modulo an affine transformation, such a network can only generate d different decision boundaries. Finally, the effect of adding more layers to the network is discussed.