# Neural Network Approximation: Three Hidden Layers Are Enough

arXiv: https://arxiv.org/abs/2010.14075 · HTML: https://mlsanity.com/paper/2010.14075/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2020.

## Built on by (most cited first, 30 of 63)

- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Operator Learning: Algorithms and Analysis](https://mlsanity.com/paper/2402.15715/) (2024)
- [Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces](https://mlsanity.com/paper/2201.00217/) (2022)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [Universal Approximation Under Constraints is Possible with Transformers](https://mlsanity.com/paper/2110.03303/) (2021)
- [Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations](https://mlsanity.com/paper/2206.09527/) (2022)
- [Stochastic Causal Programming for Bounding Treatment Effects](https://mlsanity.com/paper/2202.10806/) (2022)
- [Deep Operator Network Approximation Rates for Lipschitz Operators](https://mlsanity.com/paper/2307.09835/) (2023)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Finite Expression Method for Solving High-Dimensional Partial Differential Equations](https://mlsanity.com/paper/2206.10121/) (2022)
- [Reproducing Activation Function for Deep Learning](https://mlsanity.com/paper/2101.04844/) (2021)
- [On the Kolmogorov neural networks](https://mlsanity.com/paper/2311.00049/) (2023)
- [Neural Network Architecture Beyond Width and Depth](https://mlsanity.com/paper/2205.09459/) (2022)
- [Lyapunov-Net: A Deep Neural Network Architecture for Lyapunov Function Approximation](https://mlsanity.com/paper/2109.13359/) (2021)
- [The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation](https://mlsanity.com/paper/2103.11488/) (2021)
- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [Solving PDEs on Unknown Manifolds with Machine Learning](https://mlsanity.com/paper/2106.06682/) (2021)
- [Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networks](https://mlsanity.com/paper/2109.11354/) (2021)
- [Deep Network Approximation in Terms of Intrinsic Parameters](https://mlsanity.com/paper/2111.07964/) (2021)
- [Interpretability of Machine Learning: Recent Advances and Future Prospects](https://mlsanity.com/paper/2305.00537/) (2023)
- [Deep Neural Network Approximation of Invariant Functions through Dynamical Systems](https://mlsanity.com/paper/2208.08707/) (2022)
- [Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensions](https://mlsanity.com/paper/2312.14276/) (2023)
- [Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Class](https://mlsanity.com/paper/2103.00542/) (2021)
- [Noncompact uniform universal approximation](https://mlsanity.com/paper/2308.03812/) (2023)
- [Don't Fear Peculiar Activation Functions: EUAF and Beyond](https://mlsanity.com/paper/2407.09580/) (2024)
- [On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network](https://mlsanity.com/paper/2301.12353/) (2023)
- [Efficient Estimation in NPIV Models: A Comparison of Various Neural Networks-Based Estimators](https://mlsanity.com/paper/2110.06763/) (2021)
- [KASAM: Spline Additive Models for Function Approximation](https://mlsanity.com/paper/2205.06376/) (2022)
- [A scalable deep learning approach for solving high-dimensional dynamic optimal transport](https://mlsanity.com/paper/2205.07521/) (2022)
- [Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective](https://mlsanity.com/paper/2504.13558/) (2025)

## Abstract (arXiv)

A three-hidden-layer neural network with super approximation power is introduced. This network is built with the floor function (lfloor xrfloor), the exponential function (2x), the step function (1x≥ 0), or their compositions as the activation function in each neuron and hence we call such networks as Floor-Exponential-Step (FLES) networks. For any width hyper-parameter N∈mathbbN^+, it is shown that FLES networks with width maxd,N and three hidden layers can uniformly approximate a H"older continuous function f on [0,1]d with an exponential approximation rate 3λ (2√d)α 2-α N, where α ∈(0,1] and λ>0 are the H"older order and constant, respectively. More generally for an arbitrary continuous function f on [0,1]d with a modulus of continuity ωf(·), the constructive approximation rate is 2ωf(2√d)2-N+ωf(2√d,2-N). Moreover, we extend such a result to general bounded continuous functions on a bounded set Esubseteqℝd. As a consequence, this new class of networks overcomes the curse of dimensionality in approximation power when the variation of ωf(r) as rrightarrow 0 is moderate (e.g., ωf(r)lesssim rα for H"older continuous functions), since the major term to be concerned in our approximation rate is essentially √d times a function of N independent of d within the modulus of continuity. Finally, we extend our analysis to derive similar approximation results in the Lp-norm for p∈[1,∞) via replacing Floor-Exponential-Step activation functions by continuous activation functions.