# High-Dimensional Distribution Generation Through Deep Neural Networks

arXiv: https://arxiv.org/abs/2107.12466 · HTML: https://mlsanity.com/paper/2107.12466/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [The Power of Depth for Feedforward Neural Networks](https://mlsanity.com/paper/1512.03965/) (2015)
- [Deep Neural Network Approximation Theory](https://mlsanity.com/paper/1901.02220/) (2019)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [Nonlinear Approximation and (Deep) ReLU Networks](https://mlsanity.com/paper/1905.02199/) (2019)
- [DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text](https://mlsanity.com/paper/1802.01345/) (2018)
- [On the ability of neural nets to express distributions](https://mlsanity.com/paper/1702.07028/) (2017)
- [Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks](https://mlsanity.com/paper/2006.16664/) (2020)
- [Size-Noise Tradeoffs in Generative Networks](https://mlsanity.com/paper/1810.11158/) (2018)
- [Optimal transportation between unequal dimensions](https://mlsanity.com/paper/1805.11187/) (2018)

## Built on by (most cited first, 5 of 5)

- [Normalizing flow neural networks by JKO scheme](https://mlsanity.com/paper/2212.14424/) (2022)
- [Convergence of flow-based generative models via proximal gradient descent in Wasserstein space](https://mlsanity.com/paper/2310.17582/) (2023)
- [Approximation bounds for norm constrained neural networks with applications to regression and GANs](https://mlsanity.com/paper/2201.09418/) (2022)
- [Consistency of Neural Causal Partial Identification](https://mlsanity.com/paper/2405.15673/) (2024)
- [Learnability of high-dimensional targets by two-parameter models and gradient flow](https://mlsanity.com/paper/2402.17089/) (2024)

## Abstract (arXiv)

We show that every d-dimensional probability distribution of bounded support can be generated through deep ReLU networks out of a 1-dimensional uniform input distribution. What is more, this is possible without incurring a cost - in terms of approximation error measured in Wasserstein-distance - relative to generating the d-dimensional target distribution from d independent random variables. This is enabled by a vast generalization of the space-filling approach discovered in (Bailey & Telgarsky, 2018). The construction we propose elicits the importance of network depth in driving the Wasserstein distance between the target distribution and its neural network approximation to zero. Finally, we find that, for histogram target distributions, the number of bits needed to encode the corresponding generative network equals the fundamental limit for encoding probability distributions as dictated by quantization theory.