mlsanityData to 7 Oct 2026

Error bounds for approximations with deep ReLU networks

arXiv 1610.01145 ↗First posted 3 Oct 2016cs.LG cs.NEcited by 567

Not on the map. Why?

Abstract (arXiv)

We study expressive power of shallow and deep neural networks with piece-wise linear activation functions. We establish new rigorous upper and lower bounds for the network complexity in the setting of approximations in Sobolev spaces. In particular, we prove that deep ReLU networks more efficiently approximate smooth functions than shallow networks. In the case of approximations of 1D Lipschitz functions we describe adaptive depth-6 network architectures more efficient than the standard shallow architecture.

Key facts. Cited by 567 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Oct 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 81; 2022: 95; 2023: 96; 2024: 103; 2025: 89; 2026: 103.

567 papers.

Sort:

By area:

  1. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  2. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
  3. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  4. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Datacited by 171
  5. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  6. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  7. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  8. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  9. 2024Operator Learning: Algorithms and Analysiscited by 84
  10. 2021On the approximation of functions by tanh neural networkscited by 81
  11. 2022Neural-Fly Enables Rapid Learning for Agile Flight in Strong Windscited by 80
  12. 2023A Mathematical Guide to Operator Learningcited by 74
38 more
  1. 2022Generic bounds on the approximation error for physics-informed (and) operator learningcited by 65
  2. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  3. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  4. 2022Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spacescited by 50
  5. 2022Signal Processing for Implicit Neural Representationscited by 46
  6. 2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
  7. 2023Operator learning with PCA-Net: upper and lower complexity boundscited by 44
  8. 2021Data-driven discovery of Green's functions with human-understandable deep learningcited by 43
  9. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  10. 2024Theoretical limitations of multi-layer Transformercited by 43
  11. 2024Transformers are Universal In-context Learnerscited by 42
  12. 2021Universal Approximation Theorems for Differentiable Geometric Deep Learningcited by 41
  13. 2021Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Mapscited by 41
  14. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  15. 2022Normalizing flow neural networks by JKO schemecited by 39
  16. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  17. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Modelscited by 39
  18. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  19. 2022A Law of Data Separation in Deep Learningcited by 36
  20. 2023Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopescited by 36
  21. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  22. 2024On the expressiveness and spectral bias of KANscited by 36
  23. 2021An error analysis of generative adversarial networks for learning distributionscited by 35
  24. 2023Deep networks for system identification: a Surveycited by 35
  25. 2022Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problemscited by 34
  26. 2023Performative Prediction with Neural Networkscited by 33
  27. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  28. 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
  29. 2024GraphKAN: Enhancing Feature Extraction with Graph Kolmogorov Arnold Networkscited by 31
  30. 2021On the Existence of Universal Lottery Ticketscited by 30
  31. 2023Sumformer: Universal Approximation for Efficient Transformerscited by 30
  32. 2024Applied Causal Inference Powered by ML and AIcited by 30
  33. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  34. 2021Learning elliptic partial differential equations with randomized linear algebracited by 29
  35. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  36. 2022Nonlinear Reconstruction for Operator Learning of PDEs with Discontinuitiescited by 29
  37. 2021A Deep Learning approach to Reduced Order Modelling of Parameter Dependent Partial Differential Equationscited by 28
  38. 2022LieGG: Studying Learned Lie Group Generatorscited by 28

50 most cited of 567 All 567 (JSON)