# Functional dimension of feedforward ReLU neural networks

arXiv: https://arxiv.org/abs/2209.04036 · HTML: https://mlsanity.com/paper/2209.04036/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 8 Sep 2022.

## Builds on (most cited first, 12 of 12)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [Deep ReLU Networks Have Surprisingly Few Activation Patterns](https://mlsanity.com/paper/1906.00904/) (2019)
- [Tropical Geometry of Deep Neural Networks](https://mlsanity.com/paper/1805.07091/) (2018)
- [Reverse-Engineering Deep ReLU Networks](https://mlsanity.com/paper/1910.00744/) (2019)
- [On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks](https://mlsanity.com/paper/2008.09052/) (2020)
- [Pure and Spurious Critical Points: a Geometric Study of Linear Networks](https://mlsanity.com/paper/1910.01671/) (2019)
- [Geometry of Linear Convolutional Networks](https://mlsanity.com/paper/2108.01538/) (2021)
- [Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networks](https://mlsanity.com/paper/2207.07696/) (2022)
- [An Embedding of ReLU Networks and an Analysis of their Identifiability](https://mlsanity.com/paper/2107.09370/) (2021)
- [Local Identifiability of Deep ReLU Neural Networks: the Theory](https://mlsanity.com/paper/2206.07424/) (2022)

## Built on by (most cited first, 21 of 21)

- [Hidden symmetries of ReLU networks](https://mlsanity.com/paper/2306.06179/) (2023)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Improving Convergence and Generalization Using Parameter Symmetries](https://mlsanity.com/paper/2305.13404/) (2023)
- [Geometry of Polynomial Neural Networks](https://mlsanity.com/paper/2402.00949/) (2024)
- [Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape](https://mlsanity.com/paper/2305.19510/) (2023)
- [Topological Expressivity of ReLU Neural Networks](https://mlsanity.com/paper/2310.11130/) (2023)
- [Exploring the Complexity of Deep Neural Networks through Functional Equivalence](https://mlsanity.com/paper/2305.11417/) (2023)
- [Interpretable global minima of deep ReLU neural networks on sequentially separable data](https://mlsanity.com/paper/2405.07098/) (2024)
- [A Complete Symmetry Classification of Shallow ReLU Networks](https://mlsanity.com/paper/2604.14037/) (2026)
- [The Symmetries of Three-Layer ReLU Networks](https://mlsanity.com/paper/2605.18319/) (2026)
- [Gradient flow in parameter space is equivalent to linear interpolation in output space](https://mlsanity.com/paper/2408.01517/) (2024)
- [Teleportation With Null Space Gradient Projection for Optimization Acceleration](https://mlsanity.com/paper/2502.11362/) (2025)
- [The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networks](https://mlsanity.com/paper/2507.06367/) (2025)
- [A rank decomposition for the topological classification of neural representations](https://mlsanity.com/paper/2404.19710/) (2024)
- [On Functional Dimension and Persistent Pseudodimension](https://mlsanity.com/paper/2410.17191/) (2024)
- [Piecewise Linear Equivariant Maps for Compact Groups](https://mlsanity.com/paper/2608.21645/) (2026)
- [Stably unactivated neurons in ReLU neural networks](https://mlsanity.com/paper/2412.06829/) (2024)
- [Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning](https://mlsanity.com/paper/2501.07400/) (2025)
- [Functional Degeneracy in Neural Networks: Measurement and Pruning](https://mlsanity.com/paper/2608.30741/) (2026)
- [Hidden Activations are not Enough I: Knowledge Matrices as Higher Representations](https://mlsanity.com/paper/2609.34166/) (2026)
- [The Identifiability and Observability of Deep Normalized Attention](https://mlsanity.com/paper/2610.09620/) (2026)

## Abstract (arXiv)

It is well-known that the parameterized family of functions representable by fully-connected feedforward neural networks with ReLU activation function is precisely the class of piecewise linear functions with finitely many pieces. It is less well-known that for every fixed architecture of ReLU neural network, the parameter space admits positive-dimensional spaces of symmetries, and hence the local functional dimension near any given parameter is lower than the parametric dimension. In this work we carefully define the notion of functional dimension, show that it is inhomogeneous across the parameter space of ReLU neural network functions, and continue an investigation - initiated in [14] and [5] - into when the functional dimension achieves its theoretical maximum. We also study the quotient space and fibers of the realization map from parameter space to function space, supplying examples of fibers that are disconnected, fibers upon which functional dimension is non-constant, and fibers upon which the symmetry group acts non-transitively.