Data to 7 Oct 2026

Optimal approximation of continuous functions by very deep ReLU networks

arXiv 1802.03620 ↗First posted 10 Feb 2018cs.NEcited by 165

Not on the map. Why?

Abstract (arXiv)

We consider approximations of general continuous functions on finite-dimensional cubes by general deep ReLU neural networks and study the approximation rates with respect to the modulus of continuity of the function and the total number of weights W in the network. We establish the complete phase diagram of feasible approximation rates and show that it includes two distinct phases. One phase corresponds to slower approximations that can be achieved with constant-depth networks and continuous weight assignments. The other phase provides faster approximations at the cost of depths necessarily growing as a power law L~ Wα, 0<α≤ 1, and with necessarily discontinuous weight assignments. In particular, we prove that constant-width fully-connected networks of depth L~ W provide the fastest possible approximation rate |f-widetilde f|_∞ = O(ωf(O(W-2/nu))) that cannot be achieved with less deep networks.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 32; 2023: 29; 2024: 31; 2025: 22; 2026: 25.

165 papers.

Sort:

By area:

2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
2021On the approximation of functions by tanh neural networkscited by 81
2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
2022Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spacescited by 50
2021Evolutional Deep Neural Networkcited by 48
2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
2023Operator learning with PCA-Net: upper and lower complexity boundscited by 44
2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
2024On the expressiveness and spectral bias of KANscited by 36
2021An error analysis of generative adversarial networks for learning distributionscited by 35
2022ExSpliNet: An interpretable and expressive spline-based neural networkcited by 33

38 more

2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
2021Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spacescited by 30
2021On the Existence of Universal Lottery Ticketscited by 30
2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 28
2021On the Representation of Solutions to Elliptic PDEs in Barron Spacescited by 25
2022How do noise tails impact on deep ReLU networks?cited by 25
2021Wasserstein Generative Learning of Conditional Distributioncited by 24
2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 21
2022Most Activation Functions Can Win the Lottery Without Excessive Depthcited by 21
2021Reproducing Activation Function for Deep Learningcited by 19
2021On the capacity of deep generative networks for approximating distributionscited by 19
2021A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equationscited by 18
2023Deep Operator Learning Lessens the Curse of Dimensionality for PDEscited by 18
2022Neural Network Architecture Beyond Width and Depthcited by 17
2023Minimum width for universal approximation using ReLU networks on compact domaincited by 17
2022Convolutional and Residual Networks Provably Contain Lottery Ticketscited by 16
2024Mixtures of Neural Operators Reduce Active Complexity in Operator Learningcited by 16
2021Partition of unity networks: deep hp-approximationcited by 15
2022IAE-Net: Integral Autoencoders for Discretization-Invariant Learningcited by 15
2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
2021Universal Joint Approximation of Manifolds and Densities by Simple Injective Flowscited by 13
2021Plant 'n' Seek: Can You Find the Winning Ticket?cited by 13
2022Active Learning with Neural Networks: Insights from Nonparametric Statisticscited by 13
2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
2024On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networkscited by 13
2021Solving PDEs on Unknown Manifolds with Machine Learningcited by 12
2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
2022A general approximation lower bound in Lp norm, with applications to feed-forward neural networkscited by 12
2024On the rates of convergence for learning with convolutional neural networkscited by 10
2021A Review of Physics-based Machine Learning in Civil Engineeringcited by 9
2023Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuitscited by 9
2022Deep Neural Network Approximation of Invariant Functions through Dynamical Systemscited by 8
2023Approximation of Nonlinear Functionals Using Deep ReLU Networkscited by 8
2023Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regressioncited by 8
2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
20241-Lipschitz Neural Distance Fieldscited by 8
2021Size and Depth Separation in Approximating Benign Functions with Neural Networkscited by 7

50 most cited of 165 All 165 (JSON)