# Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

arXiv: https://arxiv.org/abs/2001.01072 · HTML: https://mlsanity.com/paper/2001.01072/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jan 2020.

## Built on by (most cited first, 30 of 36)

- [Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition](https://mlsanity.com/paper/2102.01063/) (2021)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums](https://mlsanity.com/paper/2104.08135/) (2021)
- [Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2407.09590/) (2024)
- [SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries](https://mlsanity.com/paper/2302.12828/) (2023)
- [Do Sparse Autoencoders Capture Concept Manifolds?](https://mlsanity.com/paper/2604.28119/) (2026)
- [Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networks](https://mlsanity.com/paper/2207.07696/) (2022)
- [On the Expected Complexity of Maxout Networks](https://mlsanity.com/paper/2107.00379/) (2021)
- [Are All Linear Regions Created Equal?](https://mlsanity.com/paper/2202.11749/) (2022)
- [Topology and geometry of data manifold in deep learning](https://mlsanity.com/paper/2204.08624/) (2022)
- [Origami in N dimensions: How feed-forward networks manufacture linear separability](https://mlsanity.com/paper/2203.11355/) (2022)
- [Curved Representation Space of Vision Transformers](https://mlsanity.com/paper/2210.05742/) (2022)
- [Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks](https://mlsanity.com/paper/2103.17174/) (2021)
- [Deep ReLU Networks Have Surprisingly Simple Polytopes](https://mlsanity.com/paper/2305.09145/) (2023)
- [Deep neural networks architectures from the perspective of manifold learning](https://mlsanity.com/paper/2306.03406/) (2023)
- [Neural networks behave as hash encoders: An empirical study](https://mlsanity.com/paper/2101.05490/) (2021)
- [Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics](https://mlsanity.com/paper/2301.05816/) (2023)
- [Generalization Below the Edge of Stability: The Role of Data Geometry](https://mlsanity.com/paper/2510.18120/) (2025)
- [Exploring the Common Principal Subspace of Deep Features in Neural Networks](https://mlsanity.com/paper/2110.02863/) (2021)
- [An analysis of optimization problems involving ReLU neural networks](https://mlsanity.com/paper/2502.03016/) (2025)
- [How to Explain Neural Networks: an Approximation Perspective](https://mlsanity.com/paper/2105.07831/) (2021)
- [Gradient representations in ReLU networks as similarity functions](https://mlsanity.com/paper/2110.13581/) (2021)
- [A Computational Tropical Geometry Framework for Neural Networks](https://mlsanity.com/paper/2405.20174/) (2024)
- [A Quotient Homology Theory of Representation in Neural Networks](https://mlsanity.com/paper/2502.01360/) (2025)
- [Computing Linear Regions in Neural Networks with Skip Connections](https://mlsanity.com/paper/2509.15441/) (2025)
- [Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks](https://mlsanity.com/paper/2605.04946/) (2026)
- [AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networks](https://mlsanity.com/paper/2605.06218/) (2026)
- [Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks](https://mlsanity.com/paper/2605.06300/) (2026)
- [Characterizing the Discrete Geometry of ReLU Networks](https://mlsanity.com/paper/2606.07728/) (2026)
- [SpecXAI -- Spectral interpretability of Deep Learning Models](https://mlsanity.com/paper/2302.09949/) (2023)

## Abstract (arXiv)

A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.