mlsanityData to 14 Oct 2026

Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training

arXiv 2311.18022 ↗First posted 29 Nov 2023cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

In a neural network with ReLU activations, the number of piecewise linear regions in the output can grow exponentially with depth. However, this is highly unlikely to happen when the initial parameters are sampled randomly, which therefore often leads to the use of networks that are unnecessarily large. To address this problem, we introduce a novel parameterization of the network that restricts its weights so that a depth d network produces exactly 2d linear regions at initialization and maintains those regions throughout training under the parameterization. This approach allows us to learn approximations of convex, one dimensional functions that are several orders of magnitude more accurate than their randomly initialized counterparts. We further demonstrate a preliminary extension of our construction to multidimensional and non-convex functions, allowing the technique to replace traditional dense layers in various architectures.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 42 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

24 of 42 references.

References · 24

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  7. 2016Error bounds for approximations with deep ReLU networkscited by 574
  8. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  9. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  10. 2016On the Expressive Power of Deep Neural Networkscited by 306
  11. 2020Deep Network Approximation for Smooth Functionscited by 164
  12. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
12 more
  1. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  2. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  3. 2015Representation Benefits of Deep Feedforward Networkscited by 86
  4. 2019Reverse-Engineering Deep ReLU Networkscited by 83
  5. 2018Exponential Convergence of the Deep Neural Network Approximation for Analytic Functionscited by 38
  6. 2018A comparison of deep networks with ReLU activation function and linear spline-type methodscited by 18
  7. 2019How degenerate is the parametrization of neural networks with the ReLU activation function?cited by 15
  8. 2018The universal approximation power of finite-width deep ReLU networkscited by 12
  9. 2021SERF: Towards better training of deep neural networks using log-Softplus ERror activation Functioncited by 10
  10. 2023Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networkscited by 5
  11. 2021Unsupervised Representation Learning via Neural Activation Codingcited by 4
  12. 2021Initializing ReLU networks in an expressive subspace of weightscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networkscited by 1
  2. 2026AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networkscited by 1
  3. 2026Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networkscited by 1
  4. 2026Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexitycited by 0