mlsanityData to 7 Oct 2026

Are All Linear Regions Created Equal?

arXiv 2202.11749 ↗First posted 23 Feb 2022cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

The number of linear regions has been studied as a proxy of complexity for ReLU networks. However, the empirical success of network compression techniques like pruning and knowledge distillation, suggest that in the overparameterized setting, linear regions density might fail to capture the effective nonlinearity. In this work, we propose an efficient algorithm for discovering linear regions and use it to investigate the effectiveness of density in capturing the nonlinearity of trained VGGs and ResNets on CIFAR-10 and CIFAR-100. We contrast the results with a more principled nonlinearity measure based on function variation, highlighting the shortcomings of linear regions density. Furthermore, interestingly, our measure of nonlinearity clearly correlates with model-wise deep double descent, connecting reduced test error with reduced nonlinearity, and increased local similarity of linear regions.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 23 Feb 2022.

Builds on

19 of 28 references.

References · 19

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2018On the Spectral Bias of Neural Networkscited by 1,321
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  7. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  9. 2016On the Expressive Power of Deep Neural Networkscited by 302
  10. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  11. 2016Benefits of depth in neural networkscited by 248
  12. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
7 more
  1. 2019Complexity of Linear Regions in Deep Networkscited by 166
  2. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  3. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  4. 2017Generalization in Deep Learningcited by 146
  5. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  6. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
  7. 2020Empirical Studies on the Properties of Linear Regions in Deep Neural Networkscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 3; 2025: 2; 2026: 3.

13 papers.

Sort:

  1. 2024Deep Networks Always Grok and Here is Whycited by 49
  2. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  3. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  4. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  5. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  6. 2022Deep Double Descent via Smooth Interpolationcited by 8
  7. 2025On Space Folds of ReLU Neural Networkscited by 3
  8. 2023SkelEx and BoundEx: Natural Visualization of ReLU Neural Networkscited by 0
  9. 2024Is network fragmentation a useful complexity measure?cited by 0
  10. 2025The Space Between: On Folding, Symmetries and Samplingcited by 0
  11. 2026Complexity of Linear Regions in Self-supervised Deep ReLU Networkscited by 0
  12. 2026Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexitycited by 0
1 more
  1. 2026Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulascited by 0