mlsanityData to 14 Oct 2026

Deep ReLU Networks Preserve Expected Length

arXiv 2102.10492 ↗First posted 21 Feb 2021stat.ML cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize. One natural measure of complexity is how the network distorts length - if the network takes a unit-length curve as input, what is the length of the resulting curve of outputs? It has been widely believed that this length grows exponentially in network depth. We prove that in fact this is not the case: the expected length distortion does not grow with depth, and indeed shrinks slightly, for ReLU networks with standard random initialization. We also generalize this result by proving upper bounds both for higher moments of the length distortion and for the distortion of higher-dimensional volumes. These theoretical results are corroborated by our experiments.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 26 of its 28 references that are in the map. First posted on arXiv 21 Feb 2021.

Builds on

26 of 28 references.

References · 26

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  6. 2013Do Deep Nets Really Need to be Deep?cited by 474
  7. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  9. 2016On the Expressive Power of Deep Neural Networkscited by 306
  10. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  11. 2016Benefits of depth in neural networkscited by 250
  12. 2015The Power of Depth for Feedforward Neural Networkscited by 183
14 more
  1. 2019Complexity of Linear Regions in Deep Networkscited by 167
  2. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  3. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  4. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  5. 2016Why does deep and cheap learning work so well?cited by 100
  6. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  7. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  8. 2015Representation Benefits of Deep Feedforward Networkscited by 86
  9. 2017Failures of Gradient-Based Deep Learningcited by 80
  10. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  11. 2017Depth Separation for Neural Networkscited by 46
  12. 2015Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?cited by 41
  13. 2017The power of deeper networks for expressing natural functionscited by 35
  14. 2019Trajectory growth lower bounds for random sparse deep ReLU networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 2; 2024: 1; 2025: 2; 2026: 4.

17 papers.

Sort:

  1. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  2. 2021On the Expected Complexity of Maxout Networkscited by 15
  3. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  4. 2022Lower and Upper Bounds for Numbers of Linear Regions of Graph Convolutional Networkscited by 9
  5. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  6. 2022Curved Representation Space of Vision Transformerscited by 8
  7. 2023SmoothHess: ReLU Network Feature Interactions via Stein's Lemmacited by 5
  8. 2022Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysiscited by 4
  9. 2022On Scrambling Phenomena for Randomly Initialized Recurrent Networkscited by 3
  10. 2025On Space Folds of ReLU Neural Networkscited by 3
  11. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 3
  12. 2021Gradient representations in ReLU networks as similarity functionscited by 1
5 more
  1. 2023Expected Gradients of Maxout Networks and Consequences to Parameter Initializationcited by 1
  2. 2026Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networkscited by 1
  3. 2026AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networkscited by 1
  4. 2026Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networkscited by 1
  5. 2026Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexitycited by 0