# Approximating Probability Distributions by ReLU Networks

arXiv: https://arxiv.org/abs/2101.09973 · HTML: https://mlsanity.com/paper/2101.09973/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 25 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Deep Neural Network Approximation Theory](https://mlsanity.com/paper/1901.02220/) (2019)
- [Nonlinear Approximation and (Deep) ReLU Networks](https://mlsanity.com/paper/1905.02199/) (2019)
- [On the ability of neural nets to express distributions](https://mlsanity.com/paper/1702.07028/) (2017)
- [Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks](https://mlsanity.com/paper/2006.16664/) (2020)
- [Size-Noise Tradeoffs in Generative Networks](https://mlsanity.com/paper/1810.11158/) (2018)
- [Bounds on the Approximation Power of Feedforward Neural Networks](https://mlsanity.com/paper/1806.11416/) (2018)

## Abstract (arXiv)

How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.