# Points of non-linearity of functions generated by random neural networks

arXiv: https://arxiv.org/abs/2304.09837 · HTML: https://mlsanity.com/paper/2304.09837/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 5 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 4 of 4)

- [Deep learning generalizes because the parameter-function map is biased towards simple functions](https://mlsanity.com/paper/1805.08522/) (2018)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [Is SGD a Bayesian sampler? Well, almost](https://mlsanity.com/paper/2006.15191/) (2020)
- [Neural networks are a priori biased towards Boolean functions with low entropy](https://mlsanity.com/paper/1909.11522/) (2019)

## Abstract (arXiv)

We consider functions from the real numbers to the real numbers, output by a neural network with 1 hidden activation layer, arbitrary width, and ReLU activation function. We assume that the parameters of the neural network are chosen uniformly at random with respect to various probability distributions, and compute the expected distribution of the points of non-linearity. We use these results to explain why the network may be biased towards outputting functions with simpler geometry, and why certain functions with low information-theoretic complexity are nonetheless hard for a neural network to approximate.