# The power of deeper networks for expressing natural functions

arXiv: https://arxiv.org/abs/1705.05502 · HTML: https://mlsanity.com/paper/1705.05502/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2017.

## Built on by (most cited first, 30 of 35)

- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition](https://mlsanity.com/paper/2102.01063/) (2021)
- [Fast convolutional neural networks on FPGAs with hls4ml](https://mlsanity.com/paper/2101.05108/) (2021)
- [Augmenting Deep Classifiers with Polynomial Neural Networks](https://mlsanity.com/paper/2104.07916/) (2021)
- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [Depth separation beyond radial functions](https://mlsanity.com/paper/2102.01621/) (2021)
- [Deep ReLU Networks Preserve Expected Length](https://mlsanity.com/paper/2102.10492/) (2021)
- [Explicitly antisymmetrized neural network layers for variational Monte Carlo simulation](https://mlsanity.com/paper/2112.03491/) (2021)
- [Reverse Engineering the Neural Tangent Kernel](https://mlsanity.com/paper/2106.03186/) (2021)
- [Minimal Width for Universal Property of Deep RNN](https://mlsanity.com/paper/2211.13866/) (2022)
- [Connections between Numerical Algorithms for PDEs and Neural Networks](https://mlsanity.com/paper/2107.14742/) (2021)
- [When Expressivity Meets Trainability: Fewer than n Neurons Can Work](https://mlsanity.com/paper/2210.12001/) (2022)
- [More than Correlation: Do Large Language Models Learn Causal Representations of Space?](https://mlsanity.com/paper/2312.16257/) (2023)
- [Neural Networks as Paths through the Space of Representations](https://mlsanity.com/paper/2206.10999/) (2022)
- [Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation](https://mlsanity.com/paper/2208.12401/) (2022)
- [The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks](https://mlsanity.com/paper/2310.18725/) (2023)
- [Artificial Neural Nets and the Representation of Human Concepts](https://mlsanity.com/paper/2312.05337/) (2023)
- [Depth Separation in Norm-Bounded Infinite-Width Neural Networks](https://mlsanity.com/paper/2402.08808/) (2024)
- [Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning](https://mlsanity.com/paper/2107.01081/) (2021)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [System Identification and Control Using Lyapunov-Based Deep Neural Networks without Persistent Excitation: A Concurrent Learning Approach](https://mlsanity.com/paper/2505.10678/) (2025)
- [Strategic bidding in freight transport using deep reinforcement learning](https://mlsanity.com/paper/2102.09253/) (2021)
- [Inference on Time Series Nonparametric Conditional Moment Restrictions Using General Sieves](https://mlsanity.com/paper/2301.00092/) (2022)
- [Expressivity of Shallow and Deep Neural Networks for Polynomial Approximation](https://mlsanity.com/paper/2303.03544/) (2023)
- [Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks](https://mlsanity.com/paper/2605.05659/) (2026)
- [MAGI-X: Manifold-Constrained Gaussian Process Inference for Unknown System Dynamics](https://mlsanity.com/paper/2105.12894/) (2021)
- [MOSAIC, acomparison framework for machine learning models](https://mlsanity.com/paper/2301.12986/) (2023)
- [Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss](https://mlsanity.com/paper/2309.06774/) (2023)
- [Reverse Engineering Deep ReLU Networks An Optimization-based Algorithm](https://mlsanity.com/paper/2312.04675/) (2023)

## Abstract (arXiv)

It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones. We shed light on this by proving that the total number of neurons m required to approximate natural classes of multivariate polynomials of n variables grows only linearly with n for deep neural networks, but grows exponentially when merely a single hidden layer is allowed. We also provide evidence that when the number of hidden layers is increased from 1 to k, the neuron requirement grows exponentially not with n but with n1/k, suggesting that the minimum number of layers required for practical expressibility grows only logarithmically with n.