# On the ability of neural nets to express distributions

arXiv: https://arxiv.org/abs/1702.07028 · HTML: https://mlsanity.com/paper/1702.07028/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2017.

## Built on by (most cited first, 26 of 26)

- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers](https://mlsanity.com/paper/2107.13163/) (2021)
- [The Expressive Power of Low-Rank Adaptation](https://mlsanity.com/paper/2310.17513/) (2023)
- [Normalizing flow neural networks by JKO scheme](https://mlsanity.com/paper/2212.14424/) (2022)
- [An error analysis of generative adversarial networks for learning distributions](https://mlsanity.com/paper/2105.13010/) (2021)
- [Convergence of flow-based generative models via proximal gradient descent in Wasserstein space](https://mlsanity.com/paper/2310.17582/) (2023)
- [Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks](https://mlsanity.com/paper/2109.02832/) (2021)
- [On the capacity of deep generative networks for approximating distributions](https://mlsanity.com/paper/2101.12353/) (2021)
- [Neural Network Approximations of PDEs Beyond Linearity: A Representational Perspective](https://mlsanity.com/paper/2210.12101/) (2022)
- [Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows](https://mlsanity.com/paper/2110.04227/) (2021)
- [Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian probability distributions](https://mlsanity.com/paper/2402.08082/) (2024)
- [Reliable Detection of Compressed and Encrypted Data](https://mlsanity.com/paper/2103.17059/) (2021)
- [Universal Approximation of Residual Flows in Maximum Mean Discrepancy](https://mlsanity.com/paper/2103.05793/) (2021)
- [Transformers Meet In-Context Learning: A Universal Approximation Theory](https://mlsanity.com/paper/2506.05200/) (2025)
- [High-Dimensional Distribution Generation Through Deep Neural Networks](https://mlsanity.com/paper/2107.12466/) (2021)
- [Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space](https://mlsanity.com/paper/2307.01177/) (2023)
- [Generative neural networks for characteristic functions](https://mlsanity.com/paper/2401.04778/) (2024)
- [Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem](https://mlsanity.com/paper/2110.10295/) (2021)
- [Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithms](https://mlsanity.com/paper/2209.00735/) (2022)
- [Multi-layer random features and the approximation power of neural networks](https://mlsanity.com/paper/2404.17461/) (2024)
- [Generative adversarial learning with optimal input dimension and its adaptive generator architecture](https://mlsanity.com/paper/2405.03723/) (2024)
- [Approximating Probability Distributions by ReLU Networks](https://mlsanity.com/paper/2101.09973/) (2021)
- [UnrealNAS: Can We Search Neural Architectures with Unreal Data?](https://mlsanity.com/paper/2205.02162/) (2022)
- [A method for quantifying the generalization capabilities of generative models for solving Ising models](https://mlsanity.com/paper/2405.03435/) (2024)
- [Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion](https://mlsanity.com/paper/2504.08964/) (2025)
- [Non-identifiability distinguishes Neural Networks among Parametric Models](https://mlsanity.com/paper/2504.18017/) (2025)

## Abstract (arXiv)

Deep neural nets have caused a revolution in many classification tasks. A related ongoing revolution -- also theoretically not understood -- concerns their ability to serve as generative models for complicated types of data such as images and texts. These models are trained using ideas like variational autoencoders and Generative Adversarial Networks. We take a first cut at explaining the expressivity of multilayer nets by giving a sufficient criterion for a function to be approximable by a neural network with n hidden layers. A key ingredient is Barron's Theorem citeBarron1993, which gives a Fourier criterion for approximability of a function by a neural network with 1 hidden layer. We show that a composition of n functions which satisfy certain Fourier conditions ("Barron functions") can be approximated by a n+1-layer neural network. For probability distributions, this translates into a criterion for a probability distribution to be approximable in Wasserstein distance -- a natural metric on probability distributions -- by a neural network applied to a fixed base distribution (e.g., multivariate gaussian). Building up recent lower bound work, we also give an example function that shows that composition of Barron functions is more expressive than Barron functions alone.