# Expressivity of Deep Neural Networks

arXiv: https://arxiv.org/abs/2007.04759 · HTML: https://mlsanity.com/paper/2007.04759/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2020.

## Built on by (most cited first, 30 of 32)

- [Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems](https://mlsanity.com/paper/2212.03130/) (2022)
- [Intriguing Properties of Input-dependent Randomized Smoothing](https://mlsanity.com/paper/2110.05365/) (2021)
- [On Provable Length and Compositional Generalization](https://mlsanity.com/paper/2402.04875/) (2024)
- [Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning](https://mlsanity.com/paper/2206.01342/) (2022)
- [Depth separation beyond radial functions](https://mlsanity.com/paper/2102.01621/) (2021)
- [Universal Approximation Theorem and error bounds for quantum neural networks and quantum reservoirs](https://mlsanity.com/paper/2307.12904/) (2023)
- [Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective](https://mlsanity.com/paper/2103.03113/) (2021)
- [Precision Machine Learning](https://mlsanity.com/paper/2210.13447/) (2022)
- [Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamics](https://mlsanity.com/paper/2302.09344/) (2023)
- [Deep neural network expressivity for optimal stopping problems](https://mlsanity.com/paper/2210.10443/) (2022)
- [Multilevel CNNs for Parametric PDEs](https://mlsanity.com/paper/2304.00388/) (2023)
- [Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks](https://mlsanity.com/paper/2402.05155/) (2024)
- [Size and Depth Separation in Approximating Benign Functions with Neural Networks](https://mlsanity.com/paper/2102.00314/) (2021)
- [Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks](https://mlsanity.com/paper/2503.01660/) (2025)
- [Weighted Graph-Based Signal Temporal Logic Inference Using Neural Networks](https://mlsanity.com/paper/2109.08078/) (2021)
- [Selected Trends in Artificial Intelligence for Space Applications](https://mlsanity.com/paper/2212.06662/) (2022)
- [DeNOTS: Stable Deep Neural ODEs for Time Series](https://mlsanity.com/paper/2408.08055/) (2024)
- [Approximation Theory of Tree Tensor Networks: Tensorized Multivariate Functions](https://mlsanity.com/paper/2101.11932/) (2021)
- [Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems](https://mlsanity.com/paper/2206.00934/) (2022)
- [Mean flow data assimilation using physics-constrained Graph Neural Networks](https://mlsanity.com/paper/2411.09476/) (2024)
- [Universal Approximation Property of Hamiltonian Deep Neural Networks](https://mlsanity.com/paper/2303.12147/) (2023)
- [AsyReC: A Multimodal Graph-based Framework for Spatio-Temporal Asymmetric Dyadic Relationship Classification](https://mlsanity.com/paper/2504.05030/) (2025)
- [Machine Learning Applications on Neuroimaging for Diagnosis and Prognosis of Epilepsy: A Review](https://mlsanity.com/paper/2102.03336/) (2021)
- [From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport](https://mlsanity.com/paper/2310.11439/) (2023)
- [Statistical learning by sparse deep neural networks](https://mlsanity.com/paper/2311.08845/) (2023)
- [NExG: Provable and Guided State Space Exploration of Neural Network Control Systems using Sensitivity Approximation](https://mlsanity.com/paper/2207.03884/) (2022)
- [Neural Network Predicts Ion Concentration Profiles under Nanoconfinement](https://mlsanity.com/paper/2304.04896/) (2023)
- [Deep neural network approximation of composite functions without the curse of dimensionality](https://mlsanity.com/paper/2304.05790/) (2023)
- [LiLaN: A Linear Latent Network as the Solution Operator for Real-Time Solutions to Stiff Nonlinear Ordinary Differential Equations](https://mlsanity.com/paper/2501.08423/) (2025)
- [On the algorithmic construction of deep ReLU networks](https://mlsanity.com/paper/2506.19104/) (2025)

## Abstract (arXiv)

In this review paper, we give a comprehensive overview of the large variety of approximation results for neural networks. Approximation rates for classical function spaces as well as benefits of deep neural networks over shallow ones for specifically structured function classes are discussed. While the mainbody of existing results is for general feedforward architectures, we also depict approximation results for convolutional, residual and recurrent neural networks.