mlsanityData to 7 Oct 2026

Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

arXiv 2001.01072 ↗First posted 4 Jan 2020cs.LG stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 5; 2023: 8; 2024: 3; 2025: 5; 2026: 7.

36 papers.

Sort:

By area:

  1. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  2. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  3. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  4. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  5. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  6. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
  7. 2022Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networkscited by 21
  8. 2021On the Expected Complexity of Maxout Networkscited by 15
  9. 2022Are All Linear Regions Created Equal?cited by 13
  10. 2022Topology and geometry of data manifold in deep learningcited by 12
  11. 2022Origami in N dimensions: How feed-forward networks manufacture linear separabilitycited by 9
  12. 2022Curved Representation Space of Vision Transformerscited by 8
24 more
  1. 2021Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networkscited by 7
  2. 2023Deep ReLU Networks Have Surprisingly Simple Polytopescited by 5
  3. 2023Deep neural networks architectures from the perspective of manifold learningcited by 5
  4. 2021Neural networks behave as hash encoders: An empirical studycited by 3
  5. 2023Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamicscited by 3
  6. 2021Exploring the Common Principal Subspace of Deep Features in Neural Networkscited by 2
  7. 2025An analysis of optimization problems involving ReLU neural networkscited by 2
  8. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 2
  9. 2021How to Explain Neural Networks: an Approximation Perspectivecited by 1
  10. 2021Gradient representations in ReLU networks as similarity functionscited by 1
  11. 2024A Computational Tropical Geometry Framework for Neural Networkscited by 1
  12. 2025Computing Linear Regions in Neural Networks with Skip Connectionscited by 1
  13. 2026Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networkscited by 1
  14. 2026AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networkscited by 1
  15. 2026Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networkscited by 1
  16. 2026Characterizing the Discrete Geometry of ReLU Networkscited by 1
  17. 2023SpecXAI -- Spectral interpretability of Deep Learning Modelscited by 0
  18. 2023The Power of Typed Affine Decision Structures: A Case Studycited by 0
  19. 2023SkelEx and BoundEx: Natural Visualization of ReLU Neural Networkscited by 0
  20. 2024GradAlign for Training-free Model Performance Inferencecited by 0
  21. 2025A Quotient Homology Theory of Representation in Neural Networkscited by 0
  22. 2025The Space Between: On Folding, Symmetries and Samplingcited by 0
  23. 2026Message Tuning Outshines Graph Prompt Tuning: A Prismatic Space Perspectivecited by 0
  24. 2026Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexitycited by 0