mlsanityData to 7 Oct 2026

On the Number of Linear Regions of Deep Neural Networks

arXiv 1402.1869 ↗First posted 8 Feb 2014stat.ML cs.LG cs.NEcited by 378

Not on the map. Why?

Abstract (arXiv)

We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models compute functions that react equally to complicated patterns of different inputs. The compositional structure of these functions enables them to re-use pieces of computation exponentially often in terms of the network's depth. This paper investigates the complexity of such compositional maps and contributes new theoretical results regarding the advantage of depth for neural networks with piecewise linear activation functions. In particular, our analysis is not specific to a single family of models, and as an example, we employ it for rectifier and maxout networks. We improve complexity bounds from pre-existing work and investigate the behavior of units in higher layers.

Key facts. Cited by 378 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Feb 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 68; 2022: 58; 2023: 62; 2024: 51; 2025: 72; 2026: 67.

378 papers.

Sort:

By area:

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  2. 2021Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networkscited by 219
  3. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  4. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  5. 2022B-cos Networks: Alignment is All We Need for Interpretabilitycited by 88
  6. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  7. 2022Deep Model Reassemblycited by 78
  8. 2021Robust Counterfactual Explanations on Graph Neural Networkscited by 72
  9. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  10. 2022Formal Algorithms for Transformerscited by 70
  11. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  12. 2021Model Complexity of Deep Learning: A Surveycited by 64
38 more
  1. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 64
  2. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  3. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  4. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  5. 2021Convolutional Dynamic Alignment Networks for Interpretable Classificationscited by 54
  6. 2021Single Image Defocus Deblurring Using Kernel-Sharing Parallel Atrous Convolutionscited by 54
  7. 2022Neural Architecture Search for Spiking Neural Networkscited by 52
  8. 2024Deep Networks Always Grok and Here is Whycited by 49
  9. 2022Deep Constrained Least Squares for Blind Image Super-Resolutioncited by 47
  10. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  11. 2021Discovering and Explaining the Representation Bottleneck of DNNscited by 42
  12. 2023Backpropagation-free Training of Deep Physical Neural Networkscited by 42
  13. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  14. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  15. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  16. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  17. 2021DWDN: Deep Wiener Deconvolution Network for Non-Blind Image Deblurringcited by 36
  18. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  19. 2021A Review of Formal Methods applied to Machine Learningcited by 33
  20. 2023Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theoremcited by 33
  21. 2021Neural Response Interpretation through the Lens of Critical Pathwayscited by 32
  22. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  23. 2023B-cos Alignment for Inherently Interpretable CNNs and Vision Transformerscited by 31
  24. 2021Consistent Sparse Deep Learning: Theory and Computationcited by 27
  25. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  26. 2023Can Forward Gradient Match Backpropagation?cited by 26
  27. 2021Multifidelity Modeling for Physics-Informed Neural Networks (PINNs)cited by 25
  28. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  29. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
  30. 2021MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retrainingcited by 23
  31. 2024The Real Tropical Geometry of Neural Networkscited by 23
  32. 2023Generative learning for nonlinear dynamicscited by 22
  33. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  34. 2022Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Valuescited by 21
  35. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  36. 2022Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networkscited by 21
  37. 2025Early Stopping Against Label Noise Without Validation Datacited by 21
  38. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20

50 most cited of 378 All 378 (JSON)