mlsanityData to 14 Oct 2026

On the Local Complexity of Linear Regions in Deep ReLU Networks

arXiv 2412.18283 ↗First posted 24 Dec 2024cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We define the local complexity of a neural network with continuous piecewise linear activations as a measure of the density of linear regions over an input data distribution. We show theoretically that ReLU networks that learn low-dimensional feature representations have a lower local complexity. This allows us to connect recent empirical observations on feature learning at the level of the weight matrices with concrete properties of the learned functions. In particular, we show that the local complexity serves as an upper bound on the total variation of the function over the input data distribution and thus that feature learning can be related to adversarial robustness. Lastly, we consider how optimization drives ReLU networks towards solutions with lower local complexity. Overall, this work contributes a theoretical framework towards relating geometric properties of ReLU networks to different aspects of learning such as feature learning and representation cost.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 54 of its 66 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

54 of 66 references.

References · 54

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  8. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  9. 2016On the Expressive Power of Deep Neural Networkscited by 306
  10. 2016Benefits of depth in neural networkscited by 250
  11. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  12. 2020Directional convergence and alignment in deep learningcited by 178
38 more
  1. 2019Complexity of Linear Regions in Deep Networkscited by 167
  2. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  3. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  4. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  5. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
  6. 2019How do infinite width bounded norm networks look in function space?cited by 106
  7. 2018Tropical Geometry of Deep Neural Networkscited by 95
  8. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  9. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  10. 2021Model Complexity of Deep Learning: A Surveycited by 65
  11. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  12. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  13. 2018Provable Robustness of ReLU networks via Maximization of Linear Regionscited by 51
  14. 2024Deep Networks Always Grok and Here is Whycited by 51
  15. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  16. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  17. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  18. 2022Your diffusion model secretly knows the dimension of the data manifoldcited by 36
  19. 2020On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networkscited by 35
  20. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  21. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  22. 2020On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspectivecited by 25
  23. 2024The Real Tropical Geometry of Neural Networkscited by 24
  24. 2022Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Powercited by 23
  25. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  26. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  27. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  28. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  29. 2020Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networkscited by 15
  30. 2021On the Expected Complexity of Maxout Networkscited by 15
  31. 2024Deep Grokking: Would Deep Neural Networks Generalize Better?cited by 15
  32. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  33. 2022Are All Linear Regions Created Equal?cited by 13
  34. 2024Nuclear Norm Regularization for Deep Learningcited by 11
  35. 2020Locally Linear Attributes of ReLU Neural Networkscited by 9
  36. 2021Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networkscited by 7
  37. 2022Understanding the Evolution of Linear Regions in Deep Reinforcement Learningcited by 7
  38. 2024Learning to Compress: Local Rank and Information Compression in Deep Neural Networkscited by 7

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Timecited by 4
  2. 2025Constraining the outputs of ReLU neural networkscited by 3
  3. 2026AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networkscited by 1
  4. 2026Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networkscited by 1
  5. 2026Generalization in Nonlinear Least Squares via Learned Feature Geometrycited by 1
  6. 2025ReLU Networks as Random Functions: Their Distribution in Probability Spacecited by 0
  7. 2026Geometric Decoupling: Diagnosing the Structural Instability of Latentcited by 0
  8. 2026Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexitycited by 0
  9. 2026Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulascited by 0