mlsanityData to 7 Oct 2026

The phase diagram of approximation rates for deep neural networks

arXiv 1906.09477 ↗First posted 22 Jun 2019cs.NE cs.LGcited by 86

Not on the map. Why?

Abstract (arXiv)

We explore the phase diagram of approximation rates for deep neural networks and prove several new theoretical results. In particular, we generalize the existing result on the existence of deep discontinuous phase in ReLU networks to functional classes of arbitrary positive smoothness, and identify the boundary between the feasible and infeasible rates. Moreover, we show that all networks with a piecewise polynomial activation function have the same phase diagram. Next, we demonstrate that standard fully-connected architectures with a fixed width independent of smoothness can adapt to smoothness and achieve almost optimal rates. Finally, we consider deep networks with periodic activations ("deep Fourier expansion") and prove that they have very fast, nearly exponential approximation rates, thanks to the emerging capability of the network to implement efficient lookup operations.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 16; 2023: 13; 2024: 18; 2025: 12; 2026: 10.

86 papers.

Sort:

By area:

  1. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  2. 2024Operator Learning: Algorithms and Analysiscited by 84
  3. 2022Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spacescited by 50
  4. 2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
  5. 2021Universal Approximation Theorems for Differentiable Geometric Deep Learningcited by 41
  6. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  7. 2024On the expressiveness and spectral bias of KANscited by 36
  8. 2021An error analysis of generative adversarial networks for learning distributionscited by 35
  9. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  10. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  11. 2022Achieve the Minimum Width of Neural Networks for Universal Approximationcited by 25
  12. 2023Deep Operator Network Approximation Rates for Lipschitz Operatorscited by 24
38 more
  1. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  2. 2023Distribution learning via neural differential equations: a nonparametric statistical perspectivecited by 22
  3. 2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 21
  4. 2022Finite Expression Method for Solving High-Dimensional Partial Differential Equationscited by 21
  5. 2021Reproducing Activation Function for Deep Learningcited by 19
  6. 2021On the capacity of deep generative networks for approximating distributionscited by 19
  7. 2022Neural Network Architecture Beyond Width and Depthcited by 17
  8. 2021On the rate of convergence of a classifier based on a Transformer encodercited by 16
  9. 2024Mixtures of Neural Operators Reduce Active Complexity in Operator Learningcited by 16
  10. 2024Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Losscited by 14
  11. 2022Instance-Dependent Generalization Bounds via Optimal Transportcited by 13
  12. 2024On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networkscited by 13
  13. 2021Solving PDEs on Unknown Manifolds with Machine Learningcited by 12
  14. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  15. 2022A general approximation lower bound in Lp norm, with applications to feed-forward neural networkscited by 12
  16. 2021Stationary Density Estimation of Ito Diffusions Using Deep Learningcited by 9
  17. 2023Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuitscited by 9
  18. 2021Size and Depth Separation in Approximating Benign Functions with Neural Networkscited by 7
  19. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  20. 2022Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?cited by 6
  21. 2023On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcited by 6
  22. 2021Quantitative approximation results for complex-valued neural networkscited by 5
  23. 2021Estimation of a regression function on a manifold by fully connected deep neural networkscited by 5
  24. 2024Scalable Subsampling Inference for Deep Neural Networkscited by 5
  25. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  26. 2025Deep Neural Networks with General Activations: Super-Convergence in Sobolev Normscited by 5
  27. 2026Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chainingcited by 5
  28. 2021Deep neural network surrogates for non-smooth quantities of interest in shape uncertainty quantificationcited by 4
  29. 2023Deep Learning and Computational Physics (Lecture Notes)cited by 4
  30. 2025Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potentialcited by 4
  31. 2023Deep ReLU networks and high-order finite element methods II: Chebyshev emulationcited by 3
  32. 2023Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signalscited by 3
  33. 2024On the estimation rate of Bayesian PINN for inverse problemscited by 3
  34. 2025Some Super-approximation Rates of ReLU Neural Networks for Korobov Functionscited by 3
  35. 2025Learning from one graph: transductive learning guarantees via the geometry of small random worldscited by 3
  36. 2026Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networkscited by 3
  37. 2021Analysis of convolutional neural network image classifiers in a hierarchical max-pooling model with additional local poolingcited by 2
  38. 2022Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activationcited by 2

50 most cited of 86 All 86 (JSON)