mlsanityData to 7 Oct 2026

ReLU Networks as Random Functions: Their Distribution in Probability Space

arXiv 2503.22082 ↗First posted 28 Mar 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel framework for understanding trained ReLU networks as random, affine functions, where the randomness is induced by the distribution over the inputs. By characterizing the probability distribution of the network's activation patterns, we derive the discrete probability distribution over the affine functions realizable by the network. We extend this analysis to describe the probability distribution of the network's outputs. Our approach provides explicit, numerically tractable expressions for these distributions in terms of Gaussian orthant probabilities. Additionally, we develop approximation techniques to identify the support of affine functions a trained ReLU network can realize for a given distribution of inputs. Our work provides a framework for understanding the behavior and performance of ReLU networks corresponding to stochastic inputs, paving the way for more interpretable and reliable models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

21 of 33 references.

References · 21

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  3. 2016On the Expressive Power of Deep Neural Networkscited by 302
  4. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  5. 2019Complexity of Linear Regions in Deep Networkscited by 166
  6. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  7. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  8. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  9. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  10. 2015Representation Benefits of Deep Feedforward Networkscited by 84
  11. 2018Lightweight Probabilistic Deep Networkscited by 77
  12. 2018Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solutioncited by 31
9 more
  1. 2018Empirical Bounds on Linear Regions of Deep Rectifier Networkscited by 23
  2. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  3. 2017Fast MCMC sampling algorithms on polytopescited by 17
  4. 2023Exact Count of Boundary Pieces of ReLU Classifiers: Towards the Proper Complexity Measure for Classificationcited by 11
  5. 2024An Analytic Solution to Covariance Propagation in Neural Networkscited by 11
  6. 2020Deep Feature Space: A Geometrical Perspectivecited by 10
  7. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  8. 2016Estimating orthant probabilities of high dimensional Gaussian vectors with an application to set estimationcited by 3
  9. 2023Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Lifecited by 2

Built on by

Not cited yet.