# ReLU Networks as Random Functions: Their Distribution in Probability Space

arXiv: https://arxiv.org/abs/2503.22082 · HTML: https://mlsanity.com/paper/2503.22082/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Complexity of Linear Regions in Deep Networks](https://mlsanity.com/paper/1901.09021/) (2019)
- [Bounding and Counting Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1711.02114/) (2017)
- [Deep ReLU Networks Have Surprisingly Few Activation Patterns](https://mlsanity.com/paper/1906.00904/) (2019)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [On the number of response regions of deep feed forward networks with piece-wise linear activations](https://mlsanity.com/paper/1312.6098/) (2013)
- [Representation Benefits of Deep Feedforward Networks](https://mlsanity.com/paper/1509.08101/) (2015)
- [Lightweight Probabilistic Deep Networks](https://mlsanity.com/paper/1805.11327/) (2018)
- [Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution](https://mlsanity.com/paper/1802.06259/) (2018)
- [Empirical Bounds on Linear Regions of Deep Rectifier Networks](https://mlsanity.com/paper/1810.03370/) (2018)
- [On the Number of Regions of Piecewise Linear Neural Networks](https://mlsanity.com/paper/2206.08615/) (2022)
- [Fast MCMC sampling algorithms on polytopes](https://mlsanity.com/paper/1710.08165/) (2017)
- [An Analytic Solution to Covariance Propagation in Neural Networks](https://mlsanity.com/paper/2403.16163/) (2024)
- [Exact Count of Boundary Pieces of ReLU Classifiers: Towards the Proper Complexity Measure for Classification](https://mlsanity.com/paper/2306.08805/) (2023)
- [Deep Feature Space: A Geometrical Perspective](https://mlsanity.com/paper/2007.00062/) (2020)
- [On the Local Complexity of Linear Regions in Deep ReLU Networks](https://mlsanity.com/paper/2412.18283/) (2024)
- [Estimating orthant probabilities of high dimensional Gaussian vectors with an application to set estimation](https://mlsanity.com/paper/1603.05031/) (2016)
- [Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life](https://mlsanity.com/paper/2302.05818/) (2023)

## Abstract (arXiv)

This paper presents a novel framework for understanding trained ReLU networks as random, affine functions, where the randomness is induced by the distribution over the inputs. By characterizing the probability distribution of the network's activation patterns, we derive the discrete probability distribution over the affine functions realizable by the network. We extend this analysis to describe the probability distribution of the network's outputs. Our approach provides explicit, numerically tractable expressions for these distributions in terms of Gaussian orthant probabilities. Additionally, we develop approximation techniques to identify the support of affine functions a trained ReLU network can realize for a given distribution of inputs. Our work provides a framework for understanding the behavior and performance of ReLU networks corresponding to stochastic inputs, paving the way for more interpretable and reliable models.